The Essentials of Backup Rotation with nxs-backup

In the context of rapid data growth and increasing volumes of information, effective and reliable management of backups within an organization becomes a high-priority task. This is where distributed storage and backup rotation come into play — one of the key methods of optimizing the backup process. It allows for the preservation of up-to-date data by regularly creating new copies and deleting old ones.

This approach minimizes recovery time by providing multiple recovery points suitable for different scenarios. Older copies are deleted, and new ones are created according to the parameters you need, keeping the data safe from corruption. Backup rotation allows for the efficient use of disk space while ensuring relevant data is available for recovery in case of failure or loss.

Proper rotation helps maintain a balance between data availability and efficient resource use, which is an important aspect of IT infrastructure management.

nxs-backup is well-versed in backup rotation. As a backup tool, it helps create, rotate, and store backups of files and databases. Backup cannot be performed without storage, whether it's local or remote — nxs-backup can work with both.

During rotation, a potential backup will follow a certain path from its creation to being moved to storage and possibly later deleted. As mentioned, you can create various types of backups: file backups (discrete and incremental backups in GNU Tar format), physical or logical backups for MySQL/MariaDB/PostgreSQL, and backups of MongoDB and Redis.

When backing up data for, say, MongoDB, we create a dump of the database in binary format using the mongodump utility, specifying all necessary connection parameters.

job_name: mongodb
type: mongodb
tmp_dir: /var/backup/dump_tmp
safety_backup: false
deferred_copying: false
sources:
  - name: mongodb
    connect:
      db_host: localhost
      db_port: "27017"
      db_user: mongo
      db_password: mongoP@5s
    target_dbs:
      - example
    target_collections:
      - all
    exclude_dbs: []
    exclude_collections: []
    gzip: true

The exclude_dbs and exclude_connections options in the example above allow users to exclude certain collections or entire databases from the backup. This way, data that you don't need or that shouldn't be copied for security reasons will not be included in the backup.

Once we have determined what and how we want to back up, the next step is deciding where to store everything and what rotation settings to use, specifically in the storage location.

Storing a specific number of backups
The retention block by default describes the number of intervals during which backups will be stored. The example below is: 6 months, 4 weeks, and 7 days. By default, when nxs-backup runs, weekly backups are created on Sundays, and monthly backups on the 1st of each month. If it is specified to store backups for 7 days, and nxs-backup runs twice a day, the total number of backups will be 14. The mechanism works the same for weekly and monthly backups.

Additionally, you can store a specific number of backups instead of by period with the count_instead_of_period option. This will store exactly the number you specify. This is useful if, for example, you need to collect a database dump every hour in addition to regular daily backups.

All backup job configurations, except for incremental file backups, include identical storage parameters. Below is an example of the settings for storing backups in local storage.

storages_options: 
  - storage_name: local
    backup_path: /var/nxs-backup/dump
    retention:
      days: 7
      weeks: 4
      months: 6

Working with remote storage
The tool supports remote storage options such as S3, SFTP/SCP, SMB/CIFS, NFS, FTP/FTPS, and WebDav.

The connection parameters for remote storage are located in the storage_connects block of the main configuration in a list format.
Each connection contains two required parameters: a unique name and a set of connection parameters corresponding to its type.

For example, in storage_connects, the name might be s3_aws, scp_test, webdav_test, etc., followed by the corresponding parameter settings: s3_params:, scp_params:, webdav_params:

Example of using S3 remote storage and its storage parameters.

storage_connects:
- name: s3_aws
  s3_params:
    bucket_name: backups_bucket
    access_key_id: my_s3_ak_id
    secret_access_key: ENV:S3_SECRET_KEY
    endpoint: s3.amazonaws.com
    region: us-east-1

After establishing the connection, nxs-backup allows you to flexibly configure backup rotation. Let's explore the available options in more detail.

For quick recovery over a recent period, we can use several copies stored in local storage—two daily copies and one weekly copy. These copies will be regularly updated, without taking up much memory, while storing important and relevant information.

storages_options: 
  - storage_name: local
    backup_path: /var/nxs-backup/dump
    retention:
      days: 1
      weeks: 2
      months: 0

For storing backups over a longer period, we can copy dumps daily for six months by setting the rotation settings to 180/0/0. This way, we can select any needed day over a fairly long period, for example, in the event of a cyberattack, and restore critical data.

storages_options: 
  - storage_name: s3_aws
    backup_path: /backups/dump
    retention:
      days: 180
      weeks: 0
      months: 0

If we don't need that many recovery points, there's an option to set 30/0/6, where copies for the last 30 days and the last 6 months will be stored.
These are just examples of how you can use local and remote storage. You can set rotation parameters for any period and number of copies that suit you.

Disabling backup rotation
In some special cases, it may be necessary to disable backup rotation in the storage. For example, if backups are uploaded by a user without deletion rights. In this case, set the enable_rotate parameter to "false" and configure daily/weekly/monthly backups according to your requirements. It's important to remember that after disabling rotation, you will need to manually delete outdated backups to avoid overfilling the storage.

storages_options:
  - storage_name: s3
    backup_path: /backups/databases
    enable_rotate: false
    retention:
      days: 1
      weeks: 0
      months: 1

Notification methods
It would be nice if that was the end of it, but as they say, "trust but verify" has never let anyone down. During the backup creation and delivery process, various issues can arise, so it's always good to have a reliable notification channel in case of backup failure.
nxs-backup allows you not only to run multiple processes but also to monitor them with notifications on various platforms. Currently, the tool supports email and webhooks, which allow sending notifications to Telegram, Slack, Mattermost, and any other systems that accept incoming webhooks.
Notifications provide a convenient and modern way to control the backup process. Each log event has a specific level: "warning" (default), "error," "info," and "debug." You can set the minimum level of events to be sent to your notification channel. For example, an email configuration might look like this:

notifications:
  mail:
    enabled: true
    smtp_server: smtp.mail.com
    smtp_port: 465
    smtp_user: [e-mail]
    smtp_password: some5Tr0n9P@s5worD
    recipients:
      - [e-mail]
      - [e-mail]

For Slack notifications, a webhook is first created and embedded in the configuration. You can also configure events for specific levels, similar to email notifications.

notifications:
  webhooks:
  - webhook_url: https://hooks.slack.com/services/T01ALFD17S5/B04AUP0DQTX/OkMtk1cq307xiiFb3rc13W
    enabled: true
    payload_message_key: "text"

Telegram
Notifications in Telegram are set up the same way as webhooks.

notifications:
  webhooks:
  - webhook_url: "https://api.telegram.org/bot<bot_id>:<token>/sendMessage"
    enabled: true
    extra_headers:
      "Content-Type": "application/json"
    payload_message_key: "text"
    extra_payload:
      "chat_id": <chat_id>
      "disable_web_page_preview": 1

Custom:

notifications:
  webhooks:
  - webhook_url: "https://nxs-alert.nixys.io/v2/alert/pool"
    enabled: true
    message_level: "info"
    extra_headers:
      "X-Auth-Key": "07B2vx0l79AmPBB0OwQnqDBRIs8xL8JO1sADUE84zpWoJezlE9"
    payload_message_key: "triggerMessage"
    extra_payload:
      "isEmergencyAlert":  false
      "rawTriggerMessage": false

Conclusion
nxs-backup offers a wide range of features for creating and rotating backups for local and remote storage. This process optimizes data storage and provides flexibility in configuring various backup parameters, minimizing the risk of data loss. Notification support allows for effective monitoring of backup processes, significantly improving data storage reliability and security.
For more detailed descriptions of settings and additional features, you can refer to the official documentation or subscribe to our newsletter, where we share information on new releases, interesting use cases, and much more.
Stay updated and join the user community, where you can discuss the tool's functionality and even contribute to its development.