.NET and AWS S3 with LocalStack: How to develop with local S3 buckets

Introduction

LocalStack is an open-source framework that allows us to emulate the major AWS services locally, making it easier to develop and test cloud applications without incurring the cost and complexity of deploying to a real cloud environment.

In this post, I'll show how to configure it to emulate S3 buckets and how to interact with those buckets from a C# application.

Running LocalStack

LocalStack can be run as a CLI or using a container. In this post, I'll explain how to run it in a container with docker run and docker-compose.

ℹ️ If you don't have docker or other container runtime installed, click here for Docker installation instructions or here for Podman installation instructions.

Container

To start a container with a LocalStack instance, run the following command:

docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 -e EXTRA_CORS_ALLOWED_ORIGINS=https://app.localstack.cloud. localstack/localstack:1.3.1

Note that it is exposing port 4566 and ports 4510 to 4559 and allowing CORS access from https://app.localstack.cloud. (to allow access from the LocalStack dashboard).

🚨 It's recommended to use a specific version instead of latest to avoid problems with new versions updated automatically.

Docker compose

Starting LocalStack from docker compose is just as easy. Just add the localstack service, as below, to a docker-compose.yaml file:

version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - DEBUG=${DEBUG-}
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
      - DOCKER_HOST=unix:///var/run/docker.sock
      - EXTRA_CORS_ALLOWED_ORIGINS=https://app.localstack.cloud. # Enable access from the dashboard
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

Then run the following command:

docker-compose up

🚨 It's recommended to use a specific version instead of latest to avoid problems with new versions updated automatically.

⚠️ I've added the environment variable EXTRA_CORS_ALLOWED_ORIGINS with the value https://app.localstack.cloud. to allow access from the LocalStack dashboard.

💡 The updated docker-compose.yaml file can be found here, in the LocalStack repo.

LocalStack Dashboard

LocalStack has a web-based dashboard that allows us to manage and configure its services and visualize its logs.

Access https://app.localstack.cloud. and it will connect to the LocalStack instance running locally.

It runs in the browser, so there is no need to expose any ports to the internet.

🚨 If the dashboard says "Please start LocalStack to check the System Status" and the container log shows Blocked CORS request from forbidden origin https://app.localstack.cloud., the EXTRA_CORS_ALLOWED_ORIGINS environment variable was not correctly set to https://app.localstack.cloud.. See here.

Interacting with LocalStack using the AWS CLI

We'll use the AWS CLI to interact with LocalStack. If you don't have the AWS CLI, look here for instructions on how to install it.

Configuring a profile for LocalStack

The AWS CLI requires credentials when running. LocalStack doesn't validate the credentials by default, so will create a profile with anything as access key and secret key just to make the CLI happy.

1. In a terminal, type aws configure --profile localstack;
2. For AWS Access Key ID [None]:, type anything;
3. For AWS Secret Access Key [None]:, type anything;
4. For Default region name [None]:, type the region you prefer (for example, us-east-1);
5. For Default output format [None]:, type json.

How to create a bucket using the AWS CLI

To create a S3 bucket in LocalStack, we'll use the aws s3 mb command (mb is short for Make Bucket).

The command below will create a bucket with the name local-bucket-name using the AWS CLI profile we previously created with the name localstack. It's important to pass the --endpoint parameter or else it will try to create the bucket in AWS.

aws s3 mb s3://local-bucket-name --endpoint http://localhost:4566 --profile localstack

Looking at LocalStack dashboard we can see the bucket was created:

How to list a bucket contents using the AWS CLI

To look the contents of a bucket, we can use the aws s3 ls command:

aws s3 ls s3://local-bucket-name --endpoint http://localhost:4566 --profile localstack

Or use the LocalStack dashboard:

Accessing LocalStack from .NET

To access an S3 bucket from LocalStack in .NET we use the same libraries we use to access it in AWS.

In this example, I'll use the AWSSDK.S3 and the AWSSDK.Extensions.NETCore.Setup NuGet packages, both from AWS.

How does the AmazonS3Client get AWS access data?

When running in AWS, the AmazonS3Client will get its access data from the IAM Role attached to the service running it. When running locally, it will get from the AWS CLI profile named default or from the settings we pass to it.

In the code below, I'm checking for a configuration section with the name AWS, that is not present in the production environment. If it's found, I set the Region, ServiceURL and ForcePathStyle properties of the AmazonS3Config and pass it to the creation of the AmazonS3Client.

Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.AddAwsS3Service();

var app = builder.Build();

AwsExtensions.cs

public static class AwsExtensions
{
    public static void AddAwsS3Service(this WebApplicationBuilder builder)
    {
        if (builder.Configuration.GetSection("AWS") is null)
        {
            builder.Services.AddAWSService<IAmazonS3>();
        }
        else
        {
            builder.Services.AddSingleton<IAmazonS3>(sc =>
            {
                var awsS3Config = new AmazonS3Config
                {
                    RegionEndpoint = RegionEndpoint.GetBySystemName(builder.Configuration["AWS:Region"]),
                    ServiceURL = builder.Configuration["AWS:ServiceURL"],
                    ForcePathStyle = bool.Parse(builder.Configuration["AWS:ForcePathStyle"]!)
                };

                return new AmazonS3Client(awsS3Config);
            });
        }
    }
}

The appsettings.Development.json has the configurations pointing to the LocalStack instance:

appsettings.Development.json

{
  "BucketName": "local-bucket-name",

  "AWS": {
    "Region": "us-east-1",
    "ServiceURL": "http://localhost:4566",
    "ForcePathStyle": "true"
  }
}

⚠️ The ForcePathStyle forces the use of https://s3.amazonaws.com/<bucket-name>/<object-key> styled URLs instead of https://<bucket-name>.s3.amazonaws.com/<object-key> URLs.

The ForcePathStyle needs to be set to true for the AmazonS3Client to work with LocalStack.

Upload an image to the S3 Bucket

Using Minimal APIs, I created an endpoint that receives a file and saves it in the S3 bucket.

The code is straight forward:

Program.cs

app.MapPost("/upload", async (IAmazonS3 s3Client, IFormFile file) =>
{
    var bucketName = builder.Configuration["BucketName"]!;

    var bucketExists = await s3Client.DoesS3BucketExistAsync(bucketName);

    if (!bucketExists)
    {
        return Results.BadRequest($"Bucket {bucketName} does not exists.");
    }

    using var fileStream = file.OpenReadStream();

    var putObjectRequest = new PutObjectRequest()
    {
        BucketName = bucketName,
        Key = file.FileName,
        InputStream = fileStream
    };

    putObjectRequest.Metadata.Add("Content-Type", file.ContentType);

    var putResult = await s3Client.PutObjectAsync(putObjectRequest);

    return Results.Ok($"File {file.FileName} uploaded to S3 successfully!");
});

Now, we can test it from Postman:

Get an image from the S3 Bucket

I also created an endpoint that returns the file with the key passed by parameter from the S3 bucket:

app.MapGet("/object/{key}", async (IAmazonS3 s3Client, string key) =>
{
    var bucketName = builder.Configuration["BucketName"]!;

    var bucketExists = await s3Client.DoesS3BucketExistAsync(bucketName);

    if (!bucketExists)
    {
        return Results.BadRequest($"Bucket {bucketName} does not exists.");
    }

    try
    {
        var getObjectResponse = await s3Client.GetObjectAsync(bucketName,
            key);

        return Results.File(getObjectResponse.ResponseStream,
            getObjectResponse.Headers.ContentType);
    }
    catch (AmazonS3Exception ex) when (ex.ErrorCode.Equals("NotFound", StringComparison.OrdinalIgnoreCase))
    {
        return Results.NotFound();
    }
});

Testing from Postman, we can see the image previously uploaded:

Full source code

GitHub repository

Liked this post?

I post extra content in my personal blog. Click here to see.

Follow me

• On Feedly
• RSS
• Linkedin
• Twitter

References and Links

• LocalStack Website
• LocalStack Feature Coverage
• LocalStack Dashboard
• awslocal CLI