Couchbase Database Migration

Good Practices about the Couchbase

• Create admin level users with all permission.
• Create admin level users with just read permission, it might need if you want to collect some statistics about the couchbase.
• Create separate user for buckets or each of the bucket with limited privileges.
• Upgrading memory on the VM level is not sufficient, you need to change for each of the services in the setting present in Couchbase Console UI. Keep some memory in buffer in case if you want to run some agents like stats collector.
• As the query services uses the temporary directory to scan the result, so change it to point to path in data disk not in OS disk.
• Granting bucket level permission is not sufficient, you should grant all the permission for the bucket.
• Keep each of the services like data service, index service, query service, search service, analytic service in separate nodes. It would provide the fault tolerant incase any of the node or service crashes.
• Not sure on other database, but it doesn't support back up with query filter.
• Couchbase 5.X onwards, it supports the replica index where it's not required to define the same index in each of the nodes to provide high availability rather it can be achieved with replica index.

We performed upgrade of Couchbase 5.X to 6.X

Strategy for upgrade:

• Collect the following possible data from the previous installation:
  • CPU Core Size for the entire cluster and for each of the services
  • RAM for each of the nodes and services.
  • Data Volume.
  • Keep the specification for the VMs and Data Disks separate for each of the services as index node data disk capacity might be less than the data disk hard disk and query node doesn't use data disk significantly.
• Licensing cost for Couchbase Enterprise edition is done on node level, so make sure to use the advanced node so that more than 1 service can be accommodated if required.

Challenges with Database migrations

• No filter supports for cbbackup, we had to use the cbq script to take the query based on the timestamp and set of data.
• As the above step is not the back up actually, so we had to write the spring boot application to perform UPSERT.
  • USERT is suggested if no document in source database is updated post migration.
  • Specific field update is suggested if there is updates on both the DB.
• The above steps worked will but with some difference in meta data but can be ignored as no difference was observed when and DB operations were performed from using SDK or Couchbase Query Editor:
  • Flag: 33554432(COUCHBASE_CFFMT_JSON) vs ‘0’(COUCHBASE_VAL_IS_STRING)
  • rev, cas: value differs.

Advantages with Couchbase 6.0:

• Better UI to manage the settings and view data.
• Backup and restore with various options: There are 2 options from CB 6.0 CE to back up and restore:

Cbbackup and cbrestore

• Backup:
  • /opt/couchbase/bin/cbbackup -m full http://localhost:8091 /tmp/cb-bkp -u USER -p (The first full backup)
  • /opt/couchbase/bin/cbbackup -m diff http://localhost:8091 /tmp/cb-bkp -u USER -p PASSWORD (The next incremental backup)
• Restore
  • /opt/couchbase/bin/cbrestore /tmp/cb-bkp/2020-07-21T125200Z/2020-07-21T125200Z-full http://localhost:8091 --u USER -p PASSWORD -x rehash=1
  • /opt/couchbase/bin/cbrestore /tmp/cb-bkp/2020-07-21T125200Z/2020-07-21T125822Z-diff http://localhost:8091 -u USER -p PASSWORD -x rehash=1
• Cons:
  • While restoring, we need to restore full and then all incremental in the order.
  • Difficult to manage all incremental dirs over the longer time, only solution is have the scheduled full back and remove all previous incremental backup dirs.

Cbbackupmgr (Preferred options)

• Register archive (one time process): /opt/couchbase/bin/cbbackupmgr config -a /tmp/cb-bkp-mgr -r cluster
• Back up /opt/couchbase/bin/cbbackupmgr backup -a /tmp/cb-bkp-mgr -r cluster -c couchbase://localhost -u USER -p PASSWORD First backup will be full and then next onwards would be incremental.
• Merge all backup present under /tmp/cb-bkp-mgr/cluster/ on the day of back up /opt/couchbase/bin/cbbackupmgr merge -a /tmp/cb-bkp-mgr -r cluster --start 2020-07-21T13_43_19.471413454Z --end 2020-07-21T13_54_33.394966624Z
• Restore /opt/couchbase/bin/cbbackupmgr restore -a /tmp/cb-bkp-mgr -r cluster -c couchbase://localhost -u USER -p PASSWORD
• Pros:
  • With the merge, we can merge all backups on the day of merge so that we will have only 1 dir.
  • cbbackupmgr comes with lot of options, one of them is list to see all the backup directories.
  • /opt/couchbase/bin/cbbackupmgr list -a /tmp/cb-bkp-mgr

Learnings

• Give a thought to define bucket based on any category e.g. clients or set of clients.
• Give thought to generate the document id based on the clients or set of clients.
• Backup definition interval is business driven.