How to create a MongoDB change stream for standalone

07 August,2019 by Jack Vamvas

MongoDB Change streams is a (CDC type ?) of function allowing applications to track data changes on a MongoDB collection, db or even an entire MongoDB data implementation. A consuming  application can track the change stream and do something such as trigger an event .

When I first started looking at MongoDB  Change Streams - it was to use in conjunction with another Data streaming technology such as Apache Kafka , track the change stream, consume the data , convert and redirect to a Postgres database. Sounds simple!     

One of the benefits of implementing MongoDB  Change Stream is it  does not require tailing the oplog.From a syntax perspective - the change stream format exploits the query language and disassociates you from the oplog. 

I was under the impression you needed a full blown High Availability multi node replicaset set up, but discovered , that for development\testing purposes it is possible to initiate a replicaset. It is then possible to configure an application to  consume the change stream . In the first link below there are some sample scripts for reading a change stream 

To set up a MongoDB change stream on a standalone server the minimum requirement is MongoDB 3.6 . 

I based the implementation on these 2 links 

https://www.mongodb.com/blog/post/an-introduction-to-change-streams

https://docs.mongodb.com/manual/changeStreams/

In conjunction with researching change streams , I need to consider various design  questions.

1)Much like database triggers , would you want a process that notifies data changes to be independent of other transactional code.  I've seen data integrity and other issues related to database triggers , where over the years processes become fragmented - causing developers\DBAs to become unaware of triggers during data changes or DDL changes. 

2) Being aware of documents , collections, databases is easy to view and manage. Change streams are a different beast and require a different way of viewing

3)If an application is reliant on change stream to be updated on data changes -  can they easily be traced in a debugging situation?

 

I'm continuing research on change streams , Kafka and other data streaming methods .

 

 

 

 


Author: Jack Vamvas (http://www.dba-ninja.com)


Share:

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment on How to create a MongoDB change stream for standalone


dba-ninja.com