MongoDB incremental backups
A brief instruction of how to implement incremental backups for MongoDB capable of doing a point-in-time restore.
We recently launched a new component that is using MongoDB as its backend. We always prefer backup strategies with incremental backups for our backends, as we want to minimize data loss in case of a disaster event as much as possible.
Unfortunately, the MongoDB docs do not really cover incremental backups.
Still, I have found a few hints in the docs regarding the inclusion of oplogs in the MonogDB backups and concerning how to replay them with mongorestore, which enables you to restore entries that have occurred during the mongodump process. That encouraged us to have a deeper look into the oplogs and the possibility of developing our own backup / restore solution that supports incremental backups and point-in-time recovery.
As oplogs are used internally for synchronizing MongoDB replicas, they are only available when setting up a replica set. So, if you want to use the described technique, you first have to create a replica set (even if you are using a single mongodb instance), otherwise MongoDB will not maintain the oplog collection, which is the basis for our backup solution. If you are not familiar with replicaSets, please have a look at https://docs.mongodb.com/manual/replication/ first, as it covers all details on how to set up a replication set and how to configure your oplog properly.
To do a point-in-time recovery, you need a full backup and several incremental backups, until the point of time when your disaster event happened (the oldest transaction log has to be the first taken after the full backup).
In our environment, we make a full backup once a day, simply with `mongodump -u [USER] -p [PASSWORD] — host localhost — port 27017 — oplog — gzip -o /mnt/backups/`date \+\%Y\%m\%d_\%s`, and we do incremental backups every 15 minutes. (Keep in mind, the decision about how often to perform an incremental backup is the length of time for your potential data loss, so the timeframe concerning how often you run an incremental backup depends strongly on your needs or acceptance of data loss.)
To dump the incremental backups, we use a little helper script that extracts all operations from the oplog (and keeps us from having duplicate ops in the incremental backups).
The script is basically iterating over the incremental dumps we already created and evaluates the newest oplog entry we have already dumped with bsondump. Then, it takes that oplog entry to query all oplog entries from the oplog collection, determining which ones are newer than this entry. By using that technique, we ensure that we dump every oplog operation exactly once and have no overlapping entries in our incremental backups. It is crucially important to ensure that your oplog is big enough (the oplog is a capped collection with a fixed size), so that your oplog window is bigger than the incremental dump interval, otherwise you would lose events from one incremental dump to the next. (I strongly recommend setting up a proper monitoring plan for your oplog window.)
To restore the backup(s), we also have a little helper script in place
The script applies all the needed incremental backups on top of the given full backup (and presumes that you have already restored your full backup manually with mongorestore) to the given timestamp provided by the OPLOG_LIMIT variable. Please notice that the restore script requires the filenames of the oplog backups to be in the same format as provided by the incremental backup script for the directory name of the full backup, as mentioned above.
Disclaimer
These scripts come without warranty of any kind. Use them at your own risk. I assume no liability for the accuracy, correctness, completeness, or usefulness of any information provided by this site, or for any sort of damage using these scripts may cause.
