MySQL Backups
Taking Regular Backups of the Comet DB¶
It is important to take regular backups of your Comet DB to ensure that you can recover in the event of of any issues with your database layer.
Managed DBaaS offerings, like AWS RDS/Aurora, will generally take care of this for you, or offer an easy way to enable them.
We recommend that backups be taken at least once a day, depending on your data recovery needs.
Comet Managed Backups on K8s¶
When using an instance of MySQL deployed by a Comet Installer, or one that is not deployed in a managed service, you can use the Comet MySQL DB backup utility to take backups and upload them to an S3-compatible bucket, such as the Comet bucket, or any other of your choice.
Currently this feature is only supported in Helm managed Comet installations on Kubernetes.
Configuring Automatic Backups¶
The Comet-ML Helm Chart (starting with v4.4.0
) supports creating a CronJob which will connect to the Comet MySQL database and stream a backup (created with the mysqldump
tool) to an S3-compatible Bucket. By default, this bucket is the Comet bucket, but it can be configured to be any bucket that can be connected to with Key ID and Secret credentials.
Note: Starting with version 4.7.0
this CronJob is created by default but set to run on an impossible schedule (the 31st of February). This is to aid invoking the job ad-hoc without needing to redeploy the Helm Chart.
To enable the backup Cron Job explicitly add the following to your override values:
# ...
mysql:
# ...
backup:
enabled: true # Not necessary from version 4.7.0 on.
schedule: "0 0 * * *" # Every day at midnight
# ...
A reference for the Cron Job schedule syntax can be found here.
As stated earlier, by default this will upload the backups (in a compressed format) to the bucket used already by Comet. By default the backups will be stored under the prefix comet-db-backups
. This can be adjusted to any desired prefix like so:
# ...
mysql:
# ...
backup:
# ...
destination:
prefix: "my-cool-prefix"
# ...
You can also specify a different S3-compatible instance/bucket thusly:
# ...
mysql:
# ...
backup:
# ...
destination:
useCometBucket: false
url: "https://s3.us-east-1.amazonaws.com"
keyID: XXXX
secretKey: XXXX
bucketName: my-comet-backups
## And if you need to specify the storage class...
# storageClass: STANDARD
# ...
Backup Cleanup/Retention¶
The Cron Job will also clean up old backups, based on the specified retention days, after it completes a backup. By default, backups are retained for 30 days.
# ...
mysql:
# ...
backup:
# ...
retentionDays: 7
# ...
Running an Ad-Hoc Backup¶
A backup can be invoked on-demand using kubectl
.
kubectl create job --from=cronjob/comet-ml-mysql-backup manual-comet-db-backup`
Once the pod created for this job enters the completed
status, the backup will have been created.
It is generally a good idea to delete these manually created jobs after they complete to keep things tidy.
kubectl delete job manual-comet-db-backup
Restoring the Comet DB from a Backup¶
The method for restoring your Comet database from a backup will vary depending on how the backup was taken. Normally, in a managed DBaaS offering (like AWS RDS/Aurora) the method for restoring will be specific to the platform.
Here we will discuss restoring the Comet database from an SQL file produced by the mysqldump
tool, like those which are created by the Comet managed backup utility discussed above.
Restoring with the Comet-ML Helm Chart¶
The Comet-ML Helm Chart (starting with v4.7.0
) supports restoring the Comet database from an existing backup stored in an S3-compatible bucket, so long as this backup was created by mysqldump
and is compressed with gzip
.
Configuring Restoration¶
If your desired backup to restore from is: - Stored in the same configured for the Backups (by default the Comet bucket) - Under the configured backup prefix - Is the most recent backup present (assuming it is named with a date, such that when sorted with the sort
command it will be the last result) - Ends with the extension .sql.gz
then there is nothing that you need to configure, and you can proceed with invoking the backup.
If not, or you are in doubt, you can specify the exact backup file name to use in your override values like so:
# ...
mysql:
# ...
backup:
# ...
restore:
backupName: my-backup.sql.gz
# ...
By default the same settings used for the Backup Cron Job bucket will be used to restore from. If you want to to restore from a different S3-compatible bucket (as in the case of migrating from a different instance of self-hosted Comet) you can specify it like so:
# ...
mysql:
# ...
backup:
# ...
restore:
# ...
source:
useBackupBucketSettings: false
url: "https://s3.us-east-1.amazonaws.com"
keyID: XXXX
secretKey: XXXX
bucketName: my-other-comet-backups
prefix: comet-db-backups
## And if you need to specify the storage class...
# storageClass: STANDARD
# ...
If your backup was made by an instance of the Cron Job configured by the Helm Chart prior to version 4.7.0
or in some other way that does not include the Database create statements, you will need to also tell the restore job to handle that for you, like so:
# ...
mysql:
# ...
backup:
# ...
restore:
forceDropCreate: true
# ...
# ...
Invoking the Restore Process¶
Applying the restore configuration through Helm will not invoke the job, by design. It will exist, but be set to "suspended
".
Once you have configured the restore job, you can verify its settings by inspecting the job.
kubectl describe job comet-ml-mysql-restore
When you're ready to run the restore process simply unsuspend it.
kubectl patch job comet-ml-mysql-restore -p '{"spec":{"suspend": false}}'
WARNING: This process will DROP you existing Comet DB and restore it from the backup. If you are uncertain of the origin of the backup you are using consider creating a new backup of the current DB state first, as a safety measure.
Once you have used the restore job, we recommend deleting it so that Helm may reset it.
kubectl delete job comet-ml-mysql-restore