What's new in Postgres 12

Postgres 12 is released and being tested across different organisations. I'm researching these new Postgres features/enhancements Big focus on performance & optimisation Partitioning - Partition tables can now be referenced as foreign keys Btree - Lots of small changes, but big performance improvement , and less storage requirements for multi-column indexes Most-common-value statistics - Allow most-column-value statistics for multiple columns; previously only a single correlation value was recorded for multiple columns.This feature will allow the DBA to create multi column statistics for specific use cases Inline many CTE queries - Overcome optimization barriers Prepared plan control - Prepared statements usually... Read more →

How to create a MongoDB change stream application

In an earlier post How to create a MongoDB change stream for standalone I outlined setting up a change stream for a standalone MongoDB server. Essentially it requires initiating a Replicaset and then using the change stream feature to consume the output. When the change stream is available , a quick test is next in order. The test will print out the changes as they occur. The changes appearing will depending on the configuration. In the example below - the script is configured to monitor for changes in the db = "mydb" and collection = "mycollection". The script utilises the... Read more →

How to read MongoDB system log file

Question: How can I find and read the MongoDB system message log files? I'm troubleshooting some issues and need to gather some extra information Answer: There's a few different options for identifying and reading the messaging log file . As a first step check the "mongod.conf" file , which has reference to the systemlog file The default installation for the log file is /var/log/mongodb/mongodb.log but if you don't find it at this location , them check the mongod.conf file You'll see something like : # where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log An alternative method... Read more →

Apache Kafka FAQ

These are some common questions I've received in working with Apache Kafka. I'll keep adding more detail What is Apache Kafka ? Apache Kafka is a distributed , partitioned, replicated commit log service. Apache Kafka can also be described a Publish - Subscribe message system for distributed applications. What was the original purpose of Apache Kafka ? Apache Kafka was developed within LinkedIn. LinkedIn invested in developing a single, unified , distributed pub-sub data pipeline. Prior to Kafka - LinkedIn maintained multiple data pipelines , such as inmail messaging , site events e.g site views and other operational data. Kafka... Read more →

How to find PostgreSQL server messages logs

PostgreSQL supports several methods for logging server messages, including stderr, eventlog,csvlog and syslog. The default is to log to stderr only. This parameter can only be set in the postgresql.conf file or on the server command line. Typical situation - problem arises and you need to debug - if you are not familiar with the PostgresQL logs - you can spend some frustrating time locating the details required for debugging. Use the information to start investigating Step 1 - Identify where the log entries are sent Logon as user to use psql user=# show log_destination; log_destination ----------------- stderr (1 row)... Read more →

How to create a MongoDB change stream for standalone

MongoDB Change streams is a (CDC type ?) of function allowing applications to track data changes on a MongoDB collection, db or even an entire MongoDB data implementation. A consuming application can track the change stream and do something such as trigger an event . When I first started looking at MongoDB Change Streams - it was to use in conjunction with another Data streaming technology such as Apache Kafka , track the change stream, consume the data , convert and redirect to a Postgres database. Sounds simple! One of the benefits of implementing MongoDB Change Stream is it does... Read more →

Bad Blood - Secrets and Lies in a Silicon Valley Startup - Review

I found Bad Blood - Secrets and Lies in a Silicon Valley Startup at the local bookshop, read the synopsis and thought I'll give it a go. Three days later - completed. Rather than being a straightforward , chronological outline of the events - this book is written from different perspectives - delving into Theranos . There's technology, psychology , organisational management , legal , ethics , and impending doom aka thriller. All flowing along , the story unfolding. If you've ever worked in technology - this is a salutary tale about the requirement to ask hard questions about shiny... Read more →

Confluent Kafka and MongoDB connector

I needed to source data from a MongoDB change stream- using Kafka. The official Mongo DB connector is on https://www.confluent.io/hub/mongodb/kafka-connect-mongodb and is straightforward to install. If you're database server has an internet connection , use the confluent-hub method. Otherwise there is a manual install method - outlined in the link above. This is an example of using the confluent-hub method confluent-hub install mongodb/kafka-connect-mongodb:0.1 Once you've installed the connector stop | start the Confluent Connect confluent stop connect confluent start connect You should now see a new entry in the Connector list : MongoDBConnector For the connection to work there is... Read more →

MongoDB - WARNING: shell and server versions do not match

Question : Getting this WARNING when logging on to mongo MongoDB shell version v3.4.9 connecting to: mongodb://127.0.0.1:27017/ MongoDB server version: 3.6.12 WARNING: shell and server versions do not match I’ve restarted mongo which hasn’t fixed the issue . Can I install manually or is this something pushed down from Ops Manager ? Answer: In this situation there are two options Copy the latest binary into /usr/bin or use the latest mongo shell version. To get the information require , execute this command ps -ef | grep -i mongod ps -ef | grep -i mongod mongod 8138 1 43 May27 ?... Read more →

How to get the Java Toolbox jt400 version through Command Line

Question: How can I get the Jave Toolbox jt400 driver details through the command Line? Answer: To get the version details of the Java Toolbox jt400 version through the command line run this command - making sure you're pointing to the right jar file java -cp ./jt400.jar utilities.AboutToolbox You should see something similar to the following response IBM Toolbox for Java: Open Source Software, JTOpen 9.8, codebase 5770-SS1 V7R3M0.00 built=20190304 @Z0 Supports JDBC version 4.2 Toolbox driver version 11.8 Read more →