Which one to use - Apache Kafka versus RabbitMQ

Best case scenario (use case) for Apache Kafka 1) Requirement for partitioned order at least once write. Requirement for A to B streaming with high throughput. 2) Durable message store . Why would this be important? Consumers can get a replay of the event stream - whereas limited in message brokers - where once message is delivered , removed from queue 3) If a solution requires data stream processing. 4) Event sourcing required. Event sourcing persists the state of a business entity e.g maintaining a chain of event at different states in a sequential order for a Customer 5) More... Read more →

How to fix mongo odbc 1.1 access denied for user in Challenge and Response

When supporting developers during setting up BI Connector and the ODBC System DSN for connection to a MongoDB , a common problem is the incorrect application of the username format , leading to a "Access is Denied" error message. What should the correct username format be for the Challenge and Response method? Step 1 is setting up the BI Connector. Typically the developer initiated the command line utility mongosqld . The mongosqld command line utility accepts incoming requests from a SQL client (or ODBC DSN setup used by the SQL Client) and acts as a proxy for the incoming requests.I... Read more →

How to find Kafka port number and Zookeeper port number

Question: I need to supply the Kafka port number and Zookeeper port number for firewall exceptions. Where can I find the port number details for both Kafka and Zookeeper ? Answer : To find the Kafka port number - locate the Kafka server.properties file. Typically the server.properties file will have the information required. The file is normally found on this location . By default the port number is 9092. $(confluent_home)/etc/kafka/server.properties To find the Zookeeper port number - locate the Zookeeper properties file. Normally found on $(confluent_home)/etc/kafka/zookeeper.properties. Once you've opened the file , look for ClientPort=2181 For Zookeeper - 2181 =... Read more →

How to use Powershell export-csv without a header

Question: I'm trying to use powershell cmdlet export-csv to output a csv file - but without the header information from a sql server recordset i.e first row I'm trying to implement this powershell code snippet. $outputfile="\\myoutputfile.csv" $dt = new-object "System.Data.DataTable" --add some data into the data table dt | select * -ExcludeProperty RowError, RowState, HasErrors, Name, Table, ItemArray | export-csv -path $outputfile -NoTypeInformation Answer: It is possible to get the no header on an export-csv - but it does require a few workarounds - which seem excessive for what you're attempting to achieve I prefer to use the powershell cmdlet... Read more →

How to use curl for a RabbitMQ connection

Question : How can I use curl to test a remote connection to RabbitMQ? The goal is to use Apache Kafka as the end-goal for data streaming from RabbitMQ , but I just want to use a simple curl command line method to establish whether the credentials , host & port are valid. Note this is a remote connection Answer: If you have the RabbitMQ queue name and want to return the messages in a queue this method will get the messages. Before attempting the connection ensure the host , username,password, port are all valid. Including making sure is open... Read more →

How to import a csv file with a header into a PostgreSQL table

Question: How can I import a csv file with the first row having the column names? I want to disregard the first line Answer: Use the HEADER parameter. The HEADER parameter according to the documentation "Specifies that the file contains a header line with the names of each column in the file" This is an example of a COPY which copies the data from the csv file into the table. The HEADER parameter will force the COPY action to disregard the fist line on input COPY myschema.myTable(col1,col2,col3) FROM '/tmp/myfile1.csv' DELIMITER ',' CSV HEADER; Read more →

How to migrate MongoDB data to PostgreSQL using mongoexport

MongoDB is a non-relational database and the target database platform is relational - PostgreSQL. Is there a simple approach to migrating data from MongoDB to PostgreSQL? First problem - defining a target schema . Second problem - tweaking the target schema for any changes made by the developers on the MongoDB side aka Evolving Schema. As a one-off migration - probably not such a major problem - but if your basing a DIY ETL process on mongoexport - than it's something you'll need to consider There is a tool provided by MongoDB called mongoexport and it's capable of exporting csv... Read more →

How to get DSN list from remote servers using Powershell

Powershell comes with the SQL Server Drivers packed , so you can connect to SQL Server without setting up an ODBC DSN. This makes it easy to connect to remote SQL Servers. What if you needed to connect to other database server platforms through Powershell ? A typical method used is to set up a DSN through ODBC - utilising the drivers installed for the remote database server. As part of setting up DSNs , you may need to access remote servers and identify the current DSNs. Powershell offers a method to get the DSN list from the remote server.... Read more →

What is the Avro data format

Question:I've been specifying the Avro data format through Kafka - but I'd like to understand the format in more detail and also understand why you would use Avro . Answer: Avro is an Apache open source project which supplies data serialization. Avro supports exchanging data for Apache Hadoop. Avro is very useful in situations of different programs using different languages, exchanging data. When Avro serializes , Avro stores the data definition and the data in one file. Avro supports Schema evolution i.e data schemas changing over time is a key component of Avro. Why would you use Avro ? Once... Read more →

PostgreSQL Foreign Data Wrapper

PostgreSQL implements a feature called Foreign Data Wrapper. The feature supports creating a PostgreSQL foreign table . The foreign table acts as a proxy to a remote data source. It's a similar concept - to SQL Server Linked Server Here is an example of the basic steps to set up up a foreign data wrapper to a MongoDB. The example is for illustrative purposes. -- load extension first time after install CREATE EXTENSION mongo_fdw; -- create server object CREATE SERVER mongo_server FOREIGN DATA WRAPPER mongo_fdw OPTIONS (address 'mymongoserver.net', port '16013'); -- create user mapping CREATE USER MAPPING FOR Postgres SERVER... Read more →