The Curse of The Hour
Session management in AWS is complicated, especially when authenticating with IAM roles. A common way to obtain AWS credentials is to assume an IAM role and be given a set of temporary session keys that are only good for a certain period of time. The maximum session duration is a setting on the IAM role itself, and it is one hour by default. So if users don't specify a value for the DurationSeconds
parameter, their security credentials are valid for only one hour.
A typical boto3 request to assume IAM role looks like:
response = client.assume_role(
RoleArn='string',
RoleSessionName='string',
Policy='string',
DurationSeconds=123,
ExternalId='string',
SerialNumber='string',
TokenCode='string'
)
where DurationSeconds
is the duration of the role session. It can go up to the maximum session duration setting for the role. So if your IAM role is only setup to go up to an hour, you wouldn't be able to extend the duration of your sessions unless you update the settings on the IAM role itself.
Long-Lasting Credentials
So an alternative must be introduced to extend IAM role sessions. This is when I found RefreshableCredentials
, a botocore class acting like a container for credentials needed to authenticate requests. Moreover, it can automatically refresh the credentials! This is exactly what I need. But it's usage is poorly documented. Looking at its source code:
def __init__(self, access_key, secret_key, token, expiry_time, refresh_using, method, time_fetcher=_local_now):
The __init__
function takes several arguments, half of which I don't recognize. But there's a class method that can be used to initialize an object:
@classmethod
def create_from_metadata(cls, metadata, refresh_using, method):
instance = cls(
access_key=metadata['access_key'],
secret_key=metadata['secret_key'],
token=metadata['token'],
expiry_time=cls._expiry_datetime(metadata['expiry_time']),
method=method,
refresh_using=refresh_using)
return instance
where metadata
is a dictionary containing information abound the current session, ie. access_key
, secret_key
, token
, and expiry_time
, all are things we can get from boto3's STS client's assume_role() request.
To construct the metadata response, we make a simple boto3 API call:
import boto3
sts_client = boto3.client("sts", region_name=aws_region)
params = {
"RoleArn": self.role_name,
"RoleSessionName": self.session_name,
"DurationSeconds": 3600,
}
response = sts_client.assume_role(**params).get("Credentials")
metadata = {
"access_key": response.get("AccessKeyId"),
"secret_key": response.get("SecretAccessKey"),
"token": response.get("SessionToken"),
"expiry_time": response.get("Expiration").isoformat(),
}
refresh_using
is a callable that returns a set of new credentials, taking the format of metadata. Remember that in Python, functions are first-class citizens. You can assign them to variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions. So I just need a function that generates and returns metadata.
def _refresh(self):
" Refresh tokens by calling assume_role again "
params = {
"RoleArn": self.role_name,
"RoleSessionName": self.session_name,
"DurationSeconds": 3600,
}
response = self.sts_client.assume_role(**params).get("Credentials")
credentials = {
"access_key": response.get("AccessKeyId"),
"secret_key": response.get("SecretAccessKey"),
"token": response.get("SessionToken"),
"expiry_time": response.get("Expiration").isoformat(),
}
return credentials
Now we're ready to create a RefreshableCredentials
object:
from botocore.credentials import RefreshableCredentials
session_credentials = RefreshableCredentials.create_from_metadata(
metadata=self._refresh(),
refresh_using=self._refresh,
method="sts-assume-role",
)
and we can use the credentials to generate a IAM role session that lasts for as long as we need:
from boto3 import Session
from botocore.session import get_session
session = get_session()
session._credentials = session_credentials
session.set_config_variable("region", aws_region)
autorefresh_session = Session(botocore_session=session)
And of course we can generate a boto client within that session, ie.:
db_client = autorefresh_session.client("rds", region_name='us-east-1')
Top comments (13)
For future visitors:
RefreshableCredentials
isn't documented, but can be found in the source code.@li_chastina Thanks fir the snippets!
But one question: in order to refresh credentials do we need to call
autorefresh_session.client
each time we use a client functions or its enough to always usedb_client
reference and credentials refreshed automatically?When I use this sample and print session info, looks like its generating a new credentials on every resource request !
This post made my day.
Thanks For this great article. I am trying to use this code for S3 bucket listing and upload. I have two issues. I see that when I try to list bucket, it seems to again refresh credentials every time.
Also when trying to list bucket contents, it just gets stuck foerver without a response.
This really worked for me. You helped me a lot. Thank you Chastina!
I will post again if i find something interesting about this botocore class.
Perfect, thank you!!
Very useful
Thanks for sharing
Would you be able to share the full class code here?
Good one
Nice find and great post!