This is the next part from https://dev.to/sudo_anuj/global-disaster-recovery-for-aws-aurora-2ih8
In part one, we took backup in cross region and cross account of AWS Aurora. Now, we will test how can restore it back in case a disaster happens.
We did setup of our automated backup using Terraform but here for recovery we will use awsli which will be quick in case of data recovery.
We will use 123 as account_id for main_account and 321 for disaster_recovery account.
First, we need to note our source and destination accounts, and their regions.
Source account (GDR) and region where we have target snapshot stored in AWS backup vault: 123 (US-EAST-1)
Destination account (RCW production) and region where we will restore the Aurora database: 321 (US-WEST-2)
Note: AWS don't support single copy action that performs both cross-Region AND cross-account backup. You can choose one or the other.
For that reason, we will first copy the aurora snapshot from aws backup in a intermediate RCW production account vault-123 (US-EAST-1)
Policy updates
Before we start restoration, we need to update access policies on source and destination accounts.
Source account
Find the KMS key in use on the GDR backup vault. Update policy for that key so it can be shared with the destination account.
{
"Version": "2012-10-17",
"Id": "cab-kms-key",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::SourceAccountID :root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::SourceAccountID :root",
"arn:aws:iam::DestinationAccountID:root"
]
},
"Action": [
"kms:DescribeKey",
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey",
"kms:GenerateDataKeyWithoutPlaintext"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::SourceAccountID:root",
"arn:aws:iam::DestinationAccountID:root"
]
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
}
Destination account
We need to update AWS backup vault policy so we can copy Aurora snapshot from source destination account to destination account AWS backup vault.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::SourceAccountID:root"
},
"Action": "backup:CopyIntoBackupVault",
"Resource": "*"
}
]
}
*List AWS backup vault *
In the GDR account, we don't have AWS console access but we need to get vault name ARN that we want to check after the copy is completed.
Also, during restoration we need to get GDR vault name ARN and recovery points.
We can use below command to list recovery points in GDR vault
aws backup list-recovery-points-by-backup-vault --backup-vault-name-us-east-1-aurora-dr-backup --region us-east-1
Copy the snapshot cross-account from aws console
From the source account AWS backup vault, select AWS backup vault where our snapshot is stored.
Select the recovery point we want to restore and click actions > copy
- When you click "copy", you need to select existing destination vault ARN and IAM role with permission to copy AWS backup
Now, check the destination vault. You should see the recovery point.
Copy the snapshot cross-account from aws cli
export RECOVERY_POINT_ARN="<ARN of source vault recovery point>"
export SOURCE_VAULT_NAME="<Source Vault name>"
export TARGET_VAULT_ARN="<Destination vault ARN>"
export IAM_ROLE_ARN="<Source AWS IAM role with cross account AWS copy permission>"
export TOKEN="<customer chosen string to identify among multiple copy jobs>"
export REGION="<aws-region must be same for source and destination>"
aws backup start-copy-job --recovery-point-arn "$RECOVERY_POINT_ARN" --source-backup-vault-name "$SOURCE_VAULT_NAME" --destination-backup-vault-arn "$TARGET_VAULT_ARN" --iam-role-arn "$IAM_ROLE_ARN" --idempotency-token "$TOKEN" --region "$REGION"
Example of cross-account via aws cli from 321 eu-west-1 to 123 eu-west-1
`aws backup start-copy-job --recovery-point-arn arn:aws:rds:eu-west-1:321:cluster-snapshot:awsbackup:copyjob-32b5a35c-ec86-7957-8c41-2e26439ed634 --source-backup-vault-name 321-eu-west-1-rcw-devops-aurora-db --destination-backup-vault-arn arn:aws:backup:eu-west-1:123:backup-vault:restore-test --iam-role-arn arn:aws:iam::321:role/aws-backup-20111111421470 --idempotency-token 5aac8974-78d2-11ea-bc55-01111111 --region eu-west-1
{
"CopyJobId": "A3B03165-5D50-07C2-0289-8B887B53D8B5",
"CreationDate": 1718993105.097
}`
Note: You cannot copy Aurora snapshot cross account and cross region both at same time. Only one is possible at a time.
If you try to perform vault copy operation in different accounts and different region, you may get below error.
Copy snapshot cross-region from aws console
Similar to cross-account, we can go to source AWS vault > select recovery point ID > Actions > copy
Copy snapshot cross-region from aws cli
export RECOVERY_POINT_ARN="<ARN of source vault recovery point>"
export SOURCE_VAULT_NAME="<Source Vault name>"
export TARGET_VAULT_ARN="<Destination vault ARN>"
export IAM_ROLE_ARN="<Source AWS IAM role with cross account AWS copy permission>"
export REGION="<aws-region must be same for source and destination>"
aws backup start-copy-job --recovery-point-arn "$RECOVERY_POINT_ARN" --source-backup-vault-name "$SOURCE_VAULT_NAME" --destination-backup-vault-arn "$TARGET_VAULT_ARN" --iam-role-arn "$IAM_ROLE_ARN" --region "$REGION"
*Example execution of cross region copy operation from us-east-1 vault to us-west-2 vault *
`aws backup start-copy-job --recovery-point-arn arn:aws:rds:us-east-1:123:cluster-snapshot:awsbackup:copyjob-755c193e-8a59-ef96-9e6e --source-backup-vault-name -us-east-1-aurora-dr-backup --destination-backup-vault-arn arn:aws:backup:us-west-2:123:backup-vault:us-west-2-aurora-restore --iam-role-arn arn:aws:iam::123:role/service-role/AWSBackupDefaultServiceRole --idempotency-token 5aac8974-78d-12312313 --region us-east-1
{
"CopyJobId": "xxxxxxx-123-11111-xxxxxxxxxx5433A",
"CreationDate": 1719263869.899
}`
Restoration of database
Once we get a database snapshot in the required account and region after copy from the previous step, we can start with restoration of the database.
This will be done in 2 steps:
- Start restoration from recovery point and RDS cluster creation
- Create database instance - read and write
Start restoration from recovery point and RDS cluster creation
Go to aws account and the region where we want to restore the database. Make sure we successfully received a snapshot in the recovery point of aws backup in that region through a copy job.
Now, go to that aws vault and select recovery point we want to restore > click actions and select restore
Then, you will get the option to select before starting restoration. After selecting all options here, the Aurora cluster will start creating. This can take approx. 10 mins.
Create database instance - read and write
This step should be done after cluster creation
aws rds create-db-instance --db-instance-identifier <instance-name> --db-cluster-identifier <existing-cluster-name> --db-instance-class <db-instance-class> --engine aurora-postgresql --region <region of restoration>
References:
https://docs.aws.amazon.com/aws-backup/latest/devguide/backup-feature-availability.html#features-by-resource
https://docs.aws.amazon.com/cli/latest/reference/kms/get-key-policy.html#examples
https://repost.aws/knowledge-center/backup-troubleshoot-cross-account-copy
https://docs.aws.amazon.com/cli/latest/reference/backup/start-copy-job.html
https://docs.aws.amazon.com/aws-backup/latest/devguide/backup-feature-availability.html#features-by-resource
Top comments (0)