How to master KSQL with cheat-sheet

KSQL is the the streaming SQL Engine for the real time data processing of Kafka. These are some of the commands --check KSQL is running ps aux | grep ksql --check KSQL error logs (if using Confluent) confluent log ksql-server --list created streams LIST STREAMS --View the structure of STREAM DESCRIBE mySTREAM_NAME --display all topics SHOW TOPICS --display values in topic PRINT 'topic_name' Read more on Apache Kafka How to Rock Apache Kafka with a cheat-sheet Read more →

Which one to use - Apache Kafka versus RabbitMQ

Best case scenario (use case) for Apache Kafka 1) Requirement for partitioned order at least once write. Requirement for A to B streaming with high throughput. 2) Durable message store . Why would this be important? Consumers can get a replay of the event stream - whereas limited in message brokers - where once message is delivered , removed from queue 3) If a solution requires data stream processing. 4) Event sourcing required. Event sourcing persists the state of a business entity e.g maintaining a chain of event at different states in a sequential order for a Customer 5) More... Read more →

How to find Kafka port number and Zookeeper port number

Question: I need to supply the Kafka port number and Zookeeper port number for firewall exceptions. Where can I find the port number details for both Kafka and Zookeeper ? Answer : To find the Kafka port number - locate the Kafka server.properties file. Typically the server.properties file will have the information required. The file is normally found on this location . By default the port number is 9092. $(confluent_home)/etc/kafka/server.properties To find the Zookeeper port number - locate the Zookeeper properties file. Normally found on $(confluent_home)/etc/kafka/zookeeper.properties. Once you've opened the file , look for ClientPort=2181 For Zookeeper - 2181 =... Read more →

How to use curl for a RabbitMQ connection

Question : How can I use curl to test a remote connection to RabbitMQ? The goal is to use Apache Kafka as the end-goal for data streaming from RabbitMQ , but I just want to use a simple curl command line method to establish whether the credentials , host & port are valid. Note this is a remote connection Answer: If you have the RabbitMQ queue name and want to return the messages in a queue this method will get the messages. Before attempting the connection ensure the host , username,password, port are all valid. Including making sure is open... Read more →

What is the Avro data format

Question:I've been specifying the Avro data format through Kafka - but I'd like to understand the format in more detail and also understand why you would use Avro . Answer: Avro is an Apache open source project which supplies data serialization. Avro supports exchanging data for Apache Hadoop. Avro is very useful in situations of different programs using different languages, exchanging data. When Avro serializes , Avro stores the data definition and the data in one file. Avro supports Schema evolution i.e data schemas changing over time is a key component of Avro. Why would you use Avro ? Once... Read more →

Confluent Kafka and MongoDB connector

I needed to source data from a MongoDB change stream- using Kafka. The official Mongo DB connector is on https://www.confluent.io/hub/mongodb/kafka-connect-mongodb and is straightforward to install. If you're database server has an internet connection , use the confluent-hub method. Otherwise there is a manual install method - outlined in the link above. This is an example of using the confluent-hub method confluent-hub install mongodb/kafka-connect-mongodb:0.1 Once you've installed the connector stop | start the Confluent Connect confluent stop connect confluent start connect You should now see a new entry in the Connector list : MongoDBConnector For the connection to work there is... Read more →

How to create a log trace and send to file in Kafka

Question: How can I create a trace log file in Kafka ? I want to set the logger level to TRACE , sending the trace details to a file. Answer: To change from the current Kafka Connect logging level to the TRACE , you'll need to edit the connect-log4j.properties and restart Connect edit - $(confluent_home)/etc/kafka/connect-log4j.properties log4j.rootLogger=TRACE, traceFile, stdout log4j.appender.traceFile=org.apache.log4j.DailyRollingFileAppender log4j.appender.traceFile.DatePattern='.'yyyy-MM-dd-HH log4j.appender.traceFile.File=/tmp/kafka-connect-trace.log log4j.appender.traceFile.layout=org.apache.log4j.PatternLayout log4j.appender.traceFile.layout.ConversionPattern=[%d] %p %m (%c)%n You'll need to restart Connect for the changes to take effect Read more →

How to Rock Apache Kafka with a cheat-sheet

This Apache Kafka cheat sheet covers regularly used commands across different deployments of Apache Kafka. --view error logs on Confluent Kafka confluent log connect tail -f --How to retrieve broker properties from a client step 1 - Enter zookeeper shell zookeeper-shell localhost:2181 step 2 - get /brokers/ids/0 --Check Confluent Current confluent current --Check Confluent version confluent version --List Connectors currently loaded confluent status connectors --List Kafka topics /bin/kafka-topics --zookeeper localhost:2181 --list --Delete a Kafka topic bin/kafka-topics --zookeeper localhost:2181/chroot --delete --topic my_topic_ --Set error logging to DEBUG level edit $(confluent_home)/etc/kafka/connect-log4j.properties log4j.rootLogger=DEBUG,stdout restart Connect --List configured Connectors curl localhost:8083/connectors Read more →