Secrets Management in .NET Lambda

Typically, an application must deal with sensitive information like API keys, database credentials, etc. A recommended approach is to store this sensitive information as secrets using the Secrets Manager Service in AWS. This article will explore multiple approaches to retrieving and caching secrets in a. NET-based Lambda.

AWS SDK Based Approach

AWS provides the following package - AWSSDK.SecretsManager.Caching that can be used to retrieve and then cache the secret for future use.

To use the caching library, add the package AWSSDK.SecretsManager.Caching to your .NET Lambda

dotnet add package AWSSDK.SecretsManager.Caching

The following code snippet shows a method that retrieves a secret and cache it for 15 minutes

using Amazon.SecretsManager.Extensions.Caching;

namespace Lambda.Secrets.AWSSDK
{
    public class SecretsProvider : ISecretsProvider
    {

        //Set the cache item ttl to 15 mins
        private static SecretCacheConfiguration _cacheConfiguration = new SecretCacheConfiguration
        {
            CacheItemTTL = 900000
        };

        private SecretsManagerCache _cache = new SecretsManagerCache(_cacheConfiguration);


        public async Task<string> GetSecretAsync(string secretName)
        {
            return await _cache.GetSecretString(secretName);
        }
    }
}

To reuse a secret across the lifetime of a Lambda execution environment (i.e., while the container is warm), inject the secret-retrieving class as a singleton.

Using AddSingleton ensures the class is instantiated only once per warm environment, allowing the secret to persist across multiple invocations without re-fetching. This approach reduces overhead, improves performance, and minimizes calls to AWS Secrets Manager.

AWS Parameters and Secrets Lambda Extension

AWS provides an extension - AWS Parameters and Secrets Lambda Extension.This extension can be installed as a Lambda Layer and acts an in-memory cache for parameters and secrets.

Lambda Layers

Lambda layers provide a convenient way to manage reusable code and dependencies across multiple Lambda functions. A layer is essentially a ZIP archive that can include libraries, custom runtimes, or other necessary files. By using layers, you can avoid bundling these resources directly in your function's deployment package, reducing its size and improving maintainability.

How the Extension Works

• The extension exposes a http endpoint (http://localhost:2773) to the lambda
• When a secret is requested
  • The extension first checks the cache for an existing entry.
  • If the entry is not found, the value is retrieved from the Secrets Manager, cached with a TTL (default 300 seconds) and value is returned
  • If an entry is found, it verifies the time elapsed since the entry was added to the cache. If the elapsed time is within the configured cache TTL (time-to-live), the entry is returned from the cache, otherwise fresh data is fetched from the Secrets Manager, cached and returned.

Refer the article - Using the AWS Parameter and Secrets Lambda extension to cache parameters and secrets to understand the architecture of the extension.

Add the Layer to Your Lambda Function

• Open the AWS Lambda Console and navigate to your function.
• Under the Layers section, click Add a layer.
  • Select the option AWS layers
  • Select "AWS-Parameters-and-Secrets-Lambda-Extension" and the latest version
  • Click Add.
• The layer will be added to your Lambda

Configuring the Extension

The extension can be configured using environment variables. Some important configurations are listed below

• SECRETS_MANAGER_TTL: TTL of a secret in the cache in seconds. Must be a value e between 0 and 300. The default is 300

• PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE: The maximum number of secrets and parameters to cache. Must be a value from 0 to 1000. Default is 1000

Refer the section - AWS Parameters and Secrets Lambda Extension environment variables in the following article to get the complete list of environment variables.

The following code snippet shows a method that retrieves a secret using the extension

namespace Lambda.Secrets.Extension
{
    public class SecretsProvider : ISecretsProvider
    {
        private readonly HttpClient _httpClient;

        private readonly string GetSecretsEndpoint = "/secretsmanager/get?secretId=";

        public SecretsProvider(HttpClient httpClient)
        {
            _httpClient = httpClient;
        }

        public async Task<string> GetSecretAsync(string secretName)
        {
            var httpRequest = new HttpRequestMessage(
                HttpMethod.Get,
                new Uri($"{GetSecretsEndpoint}{HttpUtility.UrlEncode(secretName)}", UriKind.Relative));

            //Pass X-Aws-Parameters-Secrets-Token as a header. This is a required header that uses the AWS_SESSION_TOKEN value,
            //which is present in the Lambda execution environment by default. 
            httpRequest.Headers.Add("X-Aws-Parameters-Secrets-Token",
                Environment.GetEnvironmentVariable("AWS_SESSION_TOKEN")
            );
            var response = await _httpClient
                .SendAsync(httpRequest)
                .ConfigureAwait(false);

            response.EnsureSuccessStatusCode();
            var responseAsJson = await response.Content.ReadFromJsonAsync<GetSecretValueResponse>();
            return responseAsJson!.SecretString;
        }
    }
}

The class can be injected into the Lambda as given below

builder.Services.AddAWSLambdaHosting(LambdaEventSource.RestApi)
           .AddHttpClient<ISecretsProvider, SecretsProvider>(c =>
        {
            c.BaseAddress = new Uri("http://localhost:2773");
        });

Testing Approach

AWS Lambda Power Tuning was used to test the performance of the Lambda functions.

• Lambda functions were with the following memory configurations: 128 MB, 256 MB, 512 MB, 1024 MB, 2048 MB, 2560 MB,3072 MB.
• Each configuration was invoked 100 times.
• Lambda invocations were done in parallel and had a combination of cold starts and warm starts.

AWS Lambda Power Tuning measured execution time and calculated the associated costs, providing insights into performance improvements.

Test Results

The test indicates that lambda that retrieves the secret using Lambda Extension is always more performant and cheaper than the lambda that retrieves the secret using AWS SDK.

Execution Time

Memory Allocation	AWS Secrets SDK (ms)	Secrets Manager Extension (ms)
128 MB	10738	6171
256 MB	5155	3189
512 MB	1762	1391
1024 MB	934	716
1536 MB	657	191
2048 MB	398	285
2560 MB	370	310
3072 MB	446	115

Cost

Memory Allocation	AWS Secrets SDK ($)	Secrets Manager Extension ($)
128 MB	0.00002127	0.00001172
256 MB	0.00001919	0.00001089
512 MB	0.00001136	0.00000797
1024 MB	0.00000955	0.00000609
1536 MB	0.00000856	0.00000204
2048 MB	0.00000658	0.00000369
2560 MB	0.00000764	0.00000512
3072 MB	0.00001083	0.00000246

The source code for the Lambdas is shared at SecretManagementInDotnetLambda