What is the Avro data format

Question:I've been specifying the Avro data format through Kafka - but I'd like to understand the format in more detail and also understand why you would use Avro . Answer: Avro is an Apache open source project which supplies data serialization. Avro supports exchanging data for Apache Hadoop. Avro is very useful in situations of different programs using different languages, exchanging data. When Avro serializes , Avro stores the data definition and the data in one file. Avro supports Schema evolution i.e data schemas changing over time is a key component of Avro. Why would you use Avro ? Once... Read more →

Foreign Data Wrapper PostgreSQL

How do you make a foreign data wrapper? Read more →

What's new in Postgres 12

Postgres 12 is released and being tested across different organisations. I'm researching these new Postgres features/enhancements Big focus on performance & optimisation Partitioning - Partition tables can now be referenced as foreign keys Btree - Lots of small changes, but big performance improvement , and less storage requirements for multi-column indexes Most-common-value statistics - Allow most-column-value statistics for multiple columns; previously only a single correlation value was recorded for multiple columns.This feature will allow the DBA to create multi column statistics for specific use cases Inline many CTE queries - Overcome optimization barriers Prepared plan control - Prepared statements usually... Read more →

How to create a MongoDB change stream application

In an earlier post How to create a MongoDB change stream for standalone I outlined setting up a change stream for a standalone MongoDB server. Essentially it requires initiating a Replicaset and then using the change stream feature to consume the output. When the change stream is available , a quick test is next in order. The test will print out the changes as they occur. The changes appearing will depending on the configuration. In the example below - the script is configured to monitor for changes in the db = "mydb" and collection = "mycollection". The script utilises the... Read more →

How to read MongoDB system log file

Question: How can I find and read the MongoDB system message log files? I'm troubleshooting some issues and need to gather some extra information Answer: There's a few different options for identifying and reading the messaging log file . As a first step check the "mongod.conf" file , which has reference to the systemlog file The default installation for the log file is /var/log/mongodb/mongodb.log but if you don't find it at this location , them check the mongod.conf file You'll see something like : # where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log An alternative method... Read more →

Apache Kafka FAQ

These are some common questions I've received in working with Apache Kafka. I'll keep adding more detail What is Apache Kafka ? Apache Kafka is a distributed , partitioned, replicated commit log service. Apache Kafka can also be described a Publish - Subscribe message system for distributed applications. What was the original purpose of Apache Kafka ? Apache Kafka was developed within LinkedIn. LinkedIn invested in developing a single, unified , distributed pub-sub data pipeline. Prior to Kafka - LinkedIn maintained multiple data pipelines , such as inmail messaging , site events e.g site views and other operational data. Kafka... Read more →

How to find PostgreSQL server messages logs

PostgreSQL supports several methods for logging server messages, including stderr, eventlog,csvlog and syslog. The default is to log to stderr only. This parameter can only be set in the postgresql.conf file or on the server command line. Typical situation - problem arises and you need to debug - if you are not familiar with the PostgresQL logs - you can spend some frustrating time locating the details required for debugging. Use the information to start investigating Step 1 - Identify where the log entries are sent Logon as user to use psql user=# show log_destination; log_destination ----------------- stderr (1 row)... Read more →

How to create a MongoDB change stream for standalone

MongoDB Change streams is a (CDC type ?) of function allowing applications to track data changes on a MongoDB collection, db or even an entire MongoDB data implementation. A consuming application can track the change stream and do something such as trigger an event . When I first started looking at MongoDB Change Streams - it was to use in conjunction with another Data streaming technology such as Apache Kafka , track the change stream, consume the data , convert and redirect to a Postgres database. Sounds simple! One of the benefits of implementing MongoDB Change Stream is it does... Read more →

Bad Blood - Secrets and Lies in a Silicon Valley Startup - Review

I found Bad Blood - Secrets and Lies in a Silicon Valley Startup at the local bookshop, read the synopsis and thought I'll give it a go. Three days later - completed. Rather than being a straightforward , chronological outline of the events - this book is written from different perspectives - delving into Theranos . There's technology, psychology , organisational management , legal , ethics , and impending doom aka thriller. All flowing along , the story unfolding. If you've ever worked in technology - this is a salutary tale about the requirement to ask hard questions about shiny... Read more →