How to read MongoDB system log file

Question: How can I find and read the MongoDB system message log files? I'm troubleshooting some issues and need to gather some extra information Answer: There's a few different options for identifying and reading the messaging log file . As a first step check the "mongod.conf" file , which has reference to the systemlog file The default installation for the log file is /var/log/mongodb/mongodb.log but if you don't find it at this location , them check the mongod.conf file You'll see something like : # where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log An alternative method... Read more →

Apache Kafka FAQ

These are some common questions I've received in working with Apache Kafka. I'll keep adding more detail What is Apache Kafka ? Apache Kafka is a distributed , partitioned, replicated commit log service What is some common terminology used when discussing Kafka? Event Producer,Partition,Routing,Topic,Broker,Sink,Stream consumers What protocols are supported by Kafka? JSON , Avro Do you have a diagram giving a very high level view of Kafka? This diagram - gives you a flavour of possibilities. What is Zookeper? Zookeper manager distributed processes. It is compulsory part of an Apache Kafka cluster.Kafka Brokers use Zookeper for cluster controller election and... Read more →

How to find PostgreSQL server messages logs

PostgreSQL supports several methods for logging server messages, including stderr, eventlog,csvlog and syslog. The default is to log to stderr only. This parameter can only be set in the postgresql.conf file or on the server command line. Typical situation - problem arises and you need to debug - if you are not familiar with the PostgresQL logs - you can spend some frustrating time locating the details required for debugging. Use the information to start investigating Step 1 - Identify where the log entries are sent Logon as user to use psql user=# show log_destination; log_destination ----------------- stderr (1 row)... Read more →

How to create a MongoDB change stream for standalone

MongoDB Change streams is a (CDC type ?) of function allowing applications to track data changes on a MongoDB collection, db or even an entire MongoDB data implementation. A consuming application can track the change stream and do something such as trigger an event . When I first started looking at MongoDB Change Streams - it was to use in conjunction with another Data streaming technology such as Apache Kafka , track the change stream, consume the data , convert and redirect to a Postgres database. Sounds simple! One of the benefits of implementing MongoDB Change Stream is it does... Read more →

Bad Blood - Secrets and Lies in a Silicon Valley Startup - Review

I found Bad Blood - Secrets and Lies in a Silicon Valley Startup at the local bookshop, read the synopsis and thought I'll give it a go. Three days later - completed. Rather than being a straightforward , chronological outline of the events - this book is written from different perspectives - delving into Theranos . There's technology, psychology , organisational management , legal , ethics , and impending doom aka thriller. All flowing along , the story unfolding. If you've ever worked in technology - this is a salutary tale about the requirement to ask hard questions about shiny... Read more →

Confluent Kafka and MongoDB connector

I needed to source data from a MongoDB change stream- using Kafka. The official Mongo DB connector is on https://www.confluent.io/hub/mongodb/kafka-connect-mongodb and is straightforward to install. If you're database server has an internet connection , use the confluent-hub method. Otherwise there is a manual install method - outlined in the link above. This is an example of using the confluent-hub method confluent-hub install mongodb/kafka-connect-mongodb:0.1 Once you've installed the connector stop | start the Confluent Connect confluent stop connect confluent start connect You should now see a new entry in the Connector list : MongoDBConnector For the connection to work there is... Read more →

MongoDB - WARNING: shell and server versions do not match

Question : Getting this WARNING when logging on to mongo MongoDB shell version v3.4.9 connecting to: mongodb://127.0.0.1:27017/ MongoDB server version: 3.6.12 WARNING: shell and server versions do not match I’ve restarted mongo which hasn’t fixed the issue . Can I install manually or is this something pushed down from Ops Manager ? Answer: In this situation there are two options Copy the latest binary into /usr/bin or use the latest mongo shell version. ps -ef | grep -i mongod mongod 8138 1 43 May27 ? 18-05:38:54 /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-3.6.12-ent/bin/mongod -f /xtra200/mongodb/appnew_0/automation-mongod.conf option 1:Copy the latest binaries into /usr/nin cp / var/lib/mongodb-mms-automation/mongodb-linux-x86_64-3.6.12-ent/bin/* /usr/bin... Read more →

How to get the Java Toolbox jt400 version through Command Line

Question: How can I get the Jave Toolbox jt400 driver details through the command Line? Answer: To get the version details of the Java Toolbox jt400 version through the command line run this command - making sure you're pointing to the right jar file java -cp ./jt400.jar utilities.AboutToolbox You should see something similar to the following response IBM Toolbox for Java: Open Source Software, JTOpen 9.8, codebase 5770-SS1 V7R3M0.00 built=20190304 @Z0 Supports JDBC version 4.2 Toolbox driver version 11.8 Read more →

How to create a log trace and send to file in Kafka

Question: How can I create a trace log file in Kafka ? I want to set the logger level to TRACE , sending the trace details to a file. Answer: To change from the current Kafka Connect logging level to the TRACE , you'll need to edit the connect-log4j.properties and restart Connect edit - $(confluent_home)/etc/kafka/connect-log4j.properties log4j.rootLogger=TRACE, traceFile, stdout log4j.appender.traceFile=org.apache.log4j.DailyRollingFileAppender log4j.appender.traceFile.DatePattern='.'yyyy-MM-dd-HH log4j.appender.traceFile.File=/tmp/kafka-connect-trace.log log4j.appender.traceFile.layout=org.apache.log4j.PatternLayout log4j.appender.traceFile.layout.ConversionPattern=[%d] %p %m (%c)%n You'll need to restart Connect for the changes to take effect Read more →

Kafka cheatsheet

This cheatsheet covers regularly used commands across different deployments of Kafka. --view error logs on Confluent Kafka confluent log connect tail -f --How to retrieve broker properties from a client step 1 - Enter zookeeper shell zookeeper-shell localhost:2181 step 2 - get /brokers/ids/0 --Check Confluent Current confluent current --Check Confluent version confluent version --List Connectors currently loaded confluent status connectors --List Kafka topics /bin/kafka-topics --zookeeper localhost:2181 --list --Delete a Kafka topic bin/kafka-topics --zookeeper localhost:2181/chroot --delete --topic my_topic_ --Set error logging to DEBUG level edit $(confluent_home)/etc/kafka/connect-log4j.properties log4j.rootLogger=DEBUG,stdout restart Connect --List configured Connectors curl localhost:8083/connectors Read more →