Replicating objects between two S3 buckets

With no doubts achieving high availability for your architecture is a must thing nowadays in our modern world, this allows you to avoid the disaster which can cause data loss.

There are many types of replications for each component in your architecture, but we are going to discuss in this article/demo how we are going to replicate the S3.

AWS provides replication through asynchronous copying of objects across Amazon S3 buckets. Buckets that are configured for object replication can be owned by the same AWS account or by different accounts.

You can replicate objects to a single destination bucket or to multiple destination buckets. The destination buckets can be in different AWS Regions or within the same Region as the source bucket.

below is a high level design for our demo to make it more visible of how we are going to do our setup

there are pre-requisites for this architecture & we are going to do these pre-requisites in this demo, I will try to list below:

• Source S3 bucket with a public access in AZ1.
• Destination S3 bucket with a public access in AZ2.
• A role with a policy for enabling & allowing the source S3 to copy the objects to the destination bucket.
• A policy contains the replication configuration will be defined and attached to the source S3 bucket.
• A CLI with an access to AWS services (I Used CloudShell in this Demo)

Please follow the below steps in order to do our setup:
use the follow command to generate a random string to make it easier in the upcoming steps:

RANDOM_STRING=$(aws secretsmanager get-random-password \
--exclude-punctuation --exclude-uppercase \
--password-length 6 --require-each-included-type \
--output text \
--query RandomPassword)

1- We will create a source S3 bucket & enable bucket versioning through the below commands:
aws s3api create-bucket --bucket src-$RANDOM_STRING

aws s3api put-bucket-versioning \
--bucket src-$RANDOM_STRING \
--versioning-configuration Status=Enabled

2- Create a destination s3 bucket and enable bucket versioning also:

aws s3api create-bucket --bucket dst-$RANDOM_STRING

aws s3api put-bucket-versioning \
--bucket dst-$RANDOM_STRING \
--versioning-configuration Status=Enabled


3- Clone thisfile into your machine, then create a role with the premissions in the previous file using the below command:

ROLE_ARN=$(aws iam create-role --role-name S3Role \
--assume-role-policy-document file://s3-assume-role-policy.json \
--output text --query Role.Arn)

_Note: We stored the role ARN in variable ROLE_ARN.
_
4- Clone this file into your machine, this is a policy which allow the source S3 bucket to replicate files to the destination S3, we will create a role with this policy template, use the below command to replace the SRCBUCKET, DSTBUCKET in the template:

sed -e "s/DSTBUCKET/dst-${RANDOM_STRING}/g" \
-e "s|SRCBUCKET|/src-${RANDOM_STRING}|g" \
s3-perms-policy-template.json > s3-perms-policy.json

Note: a new file will be created with name s3-perms-policy.json contains the bucket names you have created earlier.

5- We will attach the policy to the role we have created earlier using the below command:

aws iam put-role-policy \
--role-name S3Role \
--policy-document file://s3-perms-policy.json \
--policy-name S3ReplicationPolicy

6- Clone this file into your machine, the file contains the replication policy and configuration, it's set to 15 minutes of replication.

7- Using the sed command to replace DSTBUCKET with your bucket name, this will end with a file called s3-replication.json:

sed -e "s|ROLEARN|${ROLE_ARN}|g" \
-e "s|DSTBUCKET|dst-${RANDOM_STRING}|g" \
s3-replication-template.json > s3-replication.json

8- Now we are going to link this file with our source S3 bucket to start the replication using the below command:

aws s3api put-bucket-replication \
--replication-configuration file://s3-replication.json \
--bucket src-${RANDOM_STRING}

You can do some validation using the below commands:

to view replication configuration:

aws s3api get-bucket-replication \
--bucket src-${RANDOM_STRING}

put a file on the source bucket and check the destination bucket:

echo "This is a test file for replication" > replication-test.txt

upload it to the s3:

aws s3 cp replication-test.txt s3://src-$RANDOM_STRING


View replication status of the file:

aws s3api head-object --bucket src-${RANDOM_STRING} \
--key replication-test.txt

Output should be like this:

{
"AcceptRanges": "bytes",
"LastModified": "2021-06-20T00:17:25+00:00",
"ContentLength": 255549,
"ETag": "\"d<<>>d\"",
"VersionId": "I<>>X",
"ContentType": "image/png",
"Metadata": {},
"ReplicationStatus": "**PENDING**"
}

*PENDING **will be turned into **COMPLETED * after the file replicated.

Reference: AWS COOKBOOK