07 August,2019 by Rambler
MongoDB Change streams is a (CDC type ?) of function allowing applications to track data changes on a MongoDB collection, db or even an entire MongoDB data implementation. A consuming application can track the change stream and do something such as trigger an event .
When I first started looking at MongoDB Change Streams - it was to use in conjunction with another Data streaming technology such as Apache Kafka , track the change stream, consume the data , convert and redirect to a Postgres database. Sounds simple!
One of the benefits of implementing MongoDB Change Stream is it does not require tailing the oplog.From a syntax perspective - the change stream format exploits the query language and disassociates you from the oplog.
I was under the impression you needed a full blown High Availability multi node replicaset set up, but discovered , that for development\testing purposes it is possible to initiate a replicaset. It is then possible to configure an application to consume the change stream . In the first link below there are some sample scripts for reading a change stream
To set up a MongoDB change stream on a standalone server the minimum requirement is MongoDB 3.6 .
I based the implementation on these 2 links
https://www.mongodb.com/blog/post/an-introduction-to-change-streams
https://docs.mongodb.com/manual/changeStreams/
In conjunction with researching change streams , I need to consider various design questions.
1)Much like database triggers , would you want a process that notifies data changes to be independent of other transactional code. I've seen data integrity and other issues related to database triggers , where over the years processes become fragmented - causing developers\DBAs to become unaware of triggers during data changes or DDL changes.
2) Being aware of documents , collections, databases is easy to view and manage. Change streams are a different beast and require a different way of viewing
3)If an application is reliant on change stream to be updated on data changes - can they easily be traced in a debugging situation?
I'm continuing research on change streams , Kafka and other data streaming methods .
This is only a preview. Your comment has not yet been posted.
As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.
Having trouble reading this image? View an alternate.
Posted by: |