How to migrate MongoDB data to PostgreSQL using mongoexport

MongoDB is a non-relational database and the target database platform is relational - PostgreSQL. Is there a simple approach to migrating data from MongoDB to PostgreSQL? First problem - defining a target schema . Second problem - tweaking the target schema for any changes made by the developers on the MongoDB side aka Evolving Schema. As a one-off migration - probably not such a major problem - but if your basing a DIY ETL process on mongoexport - than it's something you'll need to consider There is a tool provided by MongoDB called mongoexport and it's capable of exporting csv... Read more →

Apache Kafka FAQ

These are some common questions I've received in working with Apache Kafka. I'll keep adding more detail What is Apache Kafka ? Apache Kafka is a distributed , partitioned, replicated commit log service. Apache Kafka can also be described a Publish - Subscribe message system for distributed applications. What was the original purpose of Apache Kafka ? Apache Kafka was developed within LinkedIn. LinkedIn invested in developing a single, unified , distributed pub-sub data pipeline. Prior to Kafka - LinkedIn maintained multiple data pipelines , such as inmail messaging , site events e.g site views and other operational data. Kafka... Read more →

Kafka cheatsheet

This cheatsheet covers regularly used commands across different deployments of Kafka. --view error logs on Confluent Kafka confluent log connect tail -f --How to retrieve broker properties from a client step 1 - Enter zookeeper shell zookeeper-shell localhost:2181 step 2 - get /brokers/ids/0 --Check Confluent Current confluent current --Check Confluent version confluent version --List Connectors currently loaded confluent status connectors --List Kafka topics /bin/kafka-topics --zookeeper localhost:2181 --list --Delete a Kafka topic bin/kafka-topics --zookeeper localhost:2181/chroot --delete --topic my_topic_ --Set error logging to DEBUG level edit $(confluent_home)/etc/kafka/connect-log4j.properties log4j.rootLogger=DEBUG,stdout restart Connect --List configured Connectors curl localhost:8083/connectors Read more →