MongoDB Groom expanding space

During some troubleshooting during an Incident I noticed the drive space dedicated to the MongoDB  data mountpoint was filling up at an alrming rate. Upon investigation I noticed there was a Groom Job running . 

The setup was a MongoDB Blockstore configured servers. 

The purpose of MongoDB groom jobs is to remove unused blocks in blockstores and S3 blockstores .  The net result is a storage space reclaim

From the MongoDB documentation.

"FILE SIZE CHANGES DURING GROOMING

During grooming operations, you may notice that the file sizes of blockstores and S3 snapshot stores fluctuate, sometimes dramatically. This is normal during these operations."

There are two events that can automatically trigger a Groom Job .

• the percentage of living bytes drops below a given threshold and
• the number of dead bytes reaches a minimum value.

When snapshots are expired or deleted , OpsManager uses groom jobs to clean the space. the impact of the groom job is different according to the target for the MongoDB database Backups. The options are :  s3 snapshot , MongoDB Blockstore or Filesystem Snapshot 

• In S3 snapshot stores, series of steps where the blocks are copied out and source blocks deleted - leading to temporary increased disk usage 
• In MongoDB Blockstore, the groom job works by also using additional disk space up to the amount of living blocks for each job.
• In Filesystem Snapshot stores, the groom job works by deleting the expired snapshots.

Deciding on the disk space to allocate will require an understanding of the Groom Job behaviour and also the type of activity on the MongoDB backup drives.  
To monitor messages  read How to read MongoDB system log file