How to migrate hundreds of GB data from aws s3 using s3 sync in few minutes

Have you ever tried to migrate GB's of data from aws S3 to you local PC or another s3 bucket?

If so, you'll know that it's difficult because the huge size of data and small chucks of file could have taken days to do the data transfer and meanwhile new data have been piled up in the S3 Bucket . So what do you do? Many people have found solutions by simply copying the folder and files using any S3 client software manually , But there are a few things you should know before you start this data transfer activity which could save your hours & days of effort. This post will tell you what you need to know to make sure you do the data transfer in most efficient , automated way.

Before begin the activity here are few prerequisites

Prerequisites: ( aws-cli, s3 bucket, IAM user with programmatic access )

1. install the aws-cli https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html#install-msi-on-windows
2. make sure you have aws iam user with proper access read / write on the s3 buckets
3. configure the aws IAM user credentials accesskey / secretkey in the system

Once the above mentioned steps are complete to ensure everything is setup properly . open the terminal or command prompt and type the below command if every thing is good then you should be able to see the list of buckets of your account.

aws s3 ls

As quick automated way to do the data sync from aws s3 to local PC i have create the windows batch script , so you can change the value and run in your windows system. and it will do the task for you.

rem Download AWS CLI https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html#install-msi-on-windows

echo 'setting the variables'

rem source bucket name with folder ex. aws.XXXX.com.001/folder
set BUCKET_NAME=aws.yoursources3bucket.com/folder

rem destinaion directory (Local PC) name with folder ex. C:\Users\abc\Newfolder.. Do not keep space in the directory.
set DESTN_DIR=c:\data\s3

rem Access Key & Secret Key. 
set AWS_ACCESS_KEY_ID=AKIXXXX
set AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX

rem AWS CLI installation path
set AWS_PATH="C:\Program Files\Amazon\AWSCLI\bin\aws"

rem Max. requrent requests
%AWS_PATH% configure set default.s3.max_concurrent_requests 200

rem Download/Sync command
echo %BUCKET_NAME% %DESTN_DIR%
%AWS_PATH% s3 sync s3://%BUCKET_NAME% %DESTN_DIR%

Please not all the aws configurations will be store in your local PC folder in the path C:\Users\Yourname.aws

👍Follow me on GitHub, or tweet me ! here.

💟 Feel free to tweet me for any issues