A Distributed SQL database service was announced at AWS re:Invent 2024 and is currently available in a free preview. This blog post marks the beginning of a series where I will test it and provide insights into its architecture and trade-offs. You can follow me on Twitter, Bluesky, or LinkedIn to receive updates on new posts.
It's important to note that, despite being branded under the Amazon Aurora umbrella, this database shares very little in common with Amazon RDS Aurora (including its Serverless and Limitless offerings). I have detailed the panel of AWS PostgreSQL-compatible databases in the following blog post:
Currently, only two Distributed SQL databases use PostgreSQL code for SQL processing over a distributed transaction and storage engine. The first is YugabyteDB, which has been available for several years. It is open-source and compatible with PostgreSQL, meaning it can replace PostgreSQL in both new and existing applications to provide resilience and elasticity. The second is Aurora DSQL, a proprietary AWS service compatible with the PostgreSQL wire protocol and dialect. It has different runtime behavior, necessitating application design adjustments to align with its runtime behavior. Following the announcement, much has been said this week, with many wrong interpretations. This series will focus on facts: reproducible tests on the preview, available documentation, and publicly announced roadmap details.
From its home page, hit "Preview Aurora DSQL now"
Enter your AWS account Root user email address and choose an AWS account name.
In the console, look for DSQL:
You can create a cluster, a set of resources to store and access the database.
The preview is in us-east-1 region and I enable multi-region with two "linked regions": us-east-2 (around 15 milliseconds round-trip time), us-west-2 (around 60 millisecond):
Once created, two public endpoint is available, one per linked region (the third is only a witness for the quorum):
The endpoint is a DNS entry that can be used by PostgreSQL as the hostname to connect to, for example, using the PGHOST environment variable. However, the first difference with PostgreSQL is that it doesn't use the same authentication methods. Only short-lived generated passwords are allowed:
$ psql
psql: error: connection to server at "mqabtu7ic5sdfgwwf3sjstb6sy.dsql.us-east-1.on.aws" (52.21.87.57), port 5432 failed: FATAL: unable to accept connection, access denied
DETAIL: Session Id: efsohrb23wquaff3fzzf4n6waq
connection to server at "mqabtu7ic5sdfgwwf3sjstb6sy.dsql.us-east-1.on.aws" (52.21.87.57), port 5432 failed: FATAL: unable to accept connection, SSL is mandatory. AWS Authentication is required.
DETAIL: Session Id: rzsohrb4d4z3tgrij4fykqmkcy
The message "AWS Authentication is required" indicates that you need to authenticate using AWS Identity and Access Management (IAM). In the console, you can find a "Connect" button that allows you to generate an "Authentication token" for admin. This token can be used as a substitute for the PostgreSQL password for the next 15 minutes:
$ export PGHOST=mqabtu7ic5sdfgwwf3sjstb6sy.dsql.us-east-1.on.aws
$ export PGPASSWORD="mqabtu7ic5sdfgwwf3sjstb6sy.dsql.us-east-1.on.aws/?Action=DbConnectAdmin&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA3VZ74TJVMUXIP5JP%2F20241204%2Fus-east-1%2Fdsql%2Faws4_request&X-Amz-Date=20241204T002902Z&X-Amz-Expires=900&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEDkaCXVzLWVhc3QtMSJHMEUCIGuuQwrqHLv5OKMJATqkFmgfm5ErYSrfIT85uz4Dwmv1AiEA1rpEE01iLUwiCAoDJHWeDu%2FhUy%2FOq9sDfks8yp%2F4gpMq5QII4f%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw4MDI3NTYwMDg1NTQiDOQ5EspONVtjjBGaciq5AgcPeZq32Yrc5zvGpj0IU9o35A4ddB%2BgHkphWP3eCHtjQgK0Pby6ICFlT46JNWYZOpuOjS6%2FbooTci11prXT3wAEiL2p3CT52KfJdSz4uFwd4HlzZPdtzHK8kIvPjK5rGU7KnEu71nYEHIC3HqB8KATmn7nlxP%2BllK5CI80N7%2B6P2dimn4%2BH%2FJHImI11MXbmKe8GpwGyCVVgLXz9MYEa0YC8TLg3HrFWl41%2BtvZTNsgQHj02p70shXc65%2BQuHC%2FxytxMSiYvUFvSdO4vYdgp%2FNI3J20Vsf1gLusw8U8ys21Z4I68HCieZDKtoMPVEzPpG9aMxI3kfEh8FVEJB8k%2FYL9B8WzZRW%2Be0Oo11RNGmSBIvXEdQxi43fW%2FubBDSGYjDEQH1Q7yny89YkdnGvxwQ2pDvWb7UXd%2BruAwi8C8ugY6nAIxdkjvRc2B9iQDGbJtJz0%2B7Z%2BjWDD7zlvvtIsONWPdTdVCFmcyZB9xcP5VaxhWavEO2gcdhKpuQOVvjPrX6t3xttDnP69wB4jNtCijfKnkYmU8gFAE4xwpffzCDe4Q9J3stKAkpLBWG0fsOr0QPSMApkN8sWHJNYS2syhXSYxQtOMJeTVcHdI0EFQTwIoETrmKEtaA0d9uGNAhmscbqEgKFfSALmEUJ7N65xM5YcdF31JGld7qrhI9jYEDg6bM3cZyQ2gPPWcAbUpX9kTkTtiLjDIo4h8wlauIx8zubLMIjdHKMSX%2B3vf9ylpsk1VnN7d5y2Kl7xZ1eK9C0jIO5Wy9WqrEM6pZ1A0ffFjU20HYUpZRKtrwqw7zN2j12Q%3D%3D&X-Amz-Signature=ddfad709f8aa449e52b9ab041337bc851c2f7b00d4e6ffaf12c3c20a73b95547&X-Amz-SignedHeaders=host"
$ export PGUSER=admin
$ psql
psql (16.2, server 16.5)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
Type "help" for help.
postgres=> select version();
version
----------------------------------------------------------------------------------------
PostgreSQL 16.5 on aarch64-unknown-linux-gnu, compiled by clang version 19.1.3, 64-bit
(1 row)
postgres=>
This authentication has some advantages. For example, I copy pasted my commands for this blog post without trying to redact the password because it will not work anymore at the time you will read this post. However, it is the first encounter of a non-PostgreSQL compatible behavior. Aurora DSQL is made to be used with new applications built for AWS so this is probably not a problem for developers. Amazon describes how to get the identification token programmatically in the User Guide. I use the admin role that is created by default, but we can create new roles and map them with IAM users.
For my tests, I use the AWS CLI to get the PGPASSWORD with an AWS user that has the "dsql:DbConnectAdmin" action allowed (I've added the AmazonAuroraDSQLFullAccess policy).
Here is how I set the connection environment variables
export AWS_REGION=us-east-1
export PGHOST=mqabtu7ic5sdfgwwf3sjstb6sy.dsql.${AWS_REGION}.on.aws
export PGSSLMODE=require
export PGUSER=admin
export PGPASSWORD=$(
aws dsql generate-db-connect-admin-auth-token --expires-in 604800 --hostname $PGHOST | tee /dev/stderr
)
The only allowed SSL mode is "require." I defined 604800 seconds, one week, as the maximum expiration time.
$ export AWS_REGION=us-east-1
$ export PGHOST=mqabtu7ic5sdfgwwf3sjstb6sy.dsql.${AWS_REGION}.on.aws
$ export PGSSLMODE=require
$ export PGPASSWORD=$(
> aws dsql generate-db-connect-admin-auth-token --expires-in 3600 --hostname $PGHOST | tee /dev/stderr
> )
mqabtu7ic5sdfgwwf3sjstb6sy.dsql.us-east-1.on.aws/?Action=DbConnectAdmin&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3VZ74TJVCRTZLT67%2F20241206%2Fus-east-1%2Fdsql%2Faws4_request&X-Amz-Date=20241206T142832Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=c434062611bbf17bc2be9732bf7597d34b8da3a55cea00a1d2854fb7919ac4d0
$ psql
psql (16.2, server 16.5)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
Type "help" for help.
postgres=> select version();
version
----------------------------------------------------------------------------------------
PostgreSQL 16.5 on aarch64-unknown-linux-gnu, compiled by clang version 19.1.3, 64-bit
(1 row)
postgres=>
Note that there's no aurora_version() to indicate that you are connected to Aurora DSQL instead of PostgreSQL, but we will discuss the differences between PostgreSQL and Aurora DSQL in future posts.
Here is how I can know the IAM user I'm connected with:
postgres=> select sys.iam_session_user();
iam_session_user
-------------------------------------
arn:aws:iam::802756008664:user/demo
(1 row)
In the upcoming posts of this series, we will explore many features and details. All comments or questions are welcome. I'll also compare YugabyteDB, the other distributed SQL database using PostgreSQL code to highlight the difference in behavior and architecture.
For my next tests, I've created the following script to connect to one region to the other:
export AWS_REGION=$1
shift
echo "Connect to dsql.${AWS_REGION}.on.aws"
export PGHOST=$(aws dsql list-clusters --output text | awk '{print $NF ; exit }').dsql.${AWS_REGION}.on.aws
export PGSSLMODE=require
export PGUSER=admin
export PGPASSWORD=$(aws dsql generate-db-connect-admin-auth-token --expires-in 360000 --hostname $PGHOST)
psql $@
I can call dsql us-east-1 or dsql us-east-2 to run psql to one or the other region (it gets the first cluster provided by list-clusters in the region)
Top comments (0)