Fastly Compute has consistently supported developers through its SDK with strong APIs for persistence needs, such as Config, Secret, and KV Store. Recently, we introduced Fastly Object Storage, an S3-compatible storage solution, which further streamlines the process of building systems on Fastly's Edge Network. With these advancements, the question arises: how best to connect Compute to Fastly Object Storage, particularly in terms of the most efficient connection methods? Fastly Object Storage is designed for S3-compatible API access, making it possible to connect from Compute via REST api call. But what's the most efficient way to do this?
In this article, I will guide you through using the official AWS SDKs to provide straightforward and intuitive access to Fastly Object Storage's API-compatible storage from Compute.
Use the official AWS SDK with Fastly Compute
Let's dive into some concrete code examples. In this first example, we're using the Compute Go SDK, which launched its official support in the summer of last year, along with AWS's official SDK(aws/aws-sdk-go-v2) to connect with Fastly Object Storage. What makes this approach particularly effective is its use of AWS's widely-adopted standard SDK, which enables intuitive handling of S3 objects. Developers familiar with this library can quickly access it with only a few lines of code, and even those new to the SDK will find it more efficient compared to directly handling the REST API. This efficiency is due to the well-developed documentation and the type hint and suggestions offered by the SDK. Additionally, aws/aws-sdk-go-v2 is particularly convenient because it supports interfaces for other AWS services beyond S3, such as RDS and Bedrock, which means this code can be adapted for use with other AWS services, too.
import (
"context"
"fmt"
"net/http"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/fastly/compute-sdk-go/fsthttp"
)
const (
BUCKET_NAME = "YOUR_BUCKET_NAME"
ACCESS_KEY = "YOUR_ACCESS_KEY"
SECRET_KEY = "YOUR_SECRET_KEY"
)
func main() {
fsthttp.ServeFunc(func(ctx context.Context, w fsthttp.ResponseWriter, r *fsthttp.Request) {
// connect to s3
customClient := &http.Client{
Transport: fsthttp.NewTransport("origin_0"),
}
client := s3.New(s3.Options{
Region: "us-east",
BaseEndpoint: aws.String("https://us-east.object.fastlystorage.app"),
Credentials: aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(ACCESS_KEY, SECRET_KEY, "")),
UsePathStyle: true,
HTTPClient: customClient,
})
p := s3.NewListObjectsV2Paginator(client,&s3.ListObjectsV2Input{Bucket: aws.String(BUCKET_NAME)})
// generate response html
w.Header().Set("Content-Type", "text/html; charset=utf-8")
fmt.Fprintf(w, "<html><body><ul>")
for p.HasMorePages() {
page, _ := p.NextPage(context.TODO())
for _, obj := range page.Contents {
fmt.Fprintf(w, "<li>Object:%s</li>", *obj.Key)
}
}
fmt.Fprintf(w, "</ul></body></html>")
})
}
The Secret Sauce
There's one crucial implementation detail worth highlighting in our example: we're injecting a custom client as the http.Client used by the AWS SDK, as shown below:
customClient := &http.Client{
Transport: fsthttp.NewTransport("origin_0"),
}
client := s3.New(s3.Options{
...
HTTPClient: customClient,
})
When using external libraries within Fastly Compute, securing access to external system resources, such as file I/O or network operations, sometimes presents a challenge. For this scenario, the critical aspect is determining how to enable network communication. Thanks to Go's excellent abstraction in the net/http standard library, particularly the RoundTripper interface, it's relatively straightforward to ensure compatibility with external network resources. In fact, many libraries in Go are designed to allow the injection of a custom http.Client, which is advantageous in WebAssembly (WASI) environments, offering significant potential to leverage existing code assets. This adaptability is a notable strength of Go's Wasm target.
Note: To run the code demonstrated here, you'll need to configure a backend named “origin_0” with an endpoint URL in the format https://us-east.object.fastlystorage.app/ (No host override configuration is required).
What about JavaScript and Rust?
Of course we can connect with them, too! For JavaScript, in fact it’s much easier than the Go code above. All you need is to install the official AWS JS SDK by npm install @aws-sdk/client-s3 and add the following import statement at the top of the script;
import {
paginateListBuckets,
S3Client,
S3ServiceException,
} from "@aws-sdk/client-s3";
Followed by some s3 operation like below;
const client = new S3Client({
region: 'us-east',
endpoint: "https://us-east.object.fastlystorage.app",
credentials: {
accessKeyId: 'YOUR_ACCESS_KEY',
secretAccessKey:'YOUR_SECRET_KEY',
},
});
const buckets = [];
for await (const page of paginateListBuckets({ client }, {})) {
buckets.push(...page.Buckets);
}
console.log(buckets.map((bucket) => bucket.Name).join("\n"));
Lastly, in case of Rust, just like we did with Go SDK, we need a custom http client to inject to the AWS SDK for Rust to make its network layer functional with wasm32-wasi compilation target. A critical consideration for this work (the most challenging aspect) would be that the AWS SDK for Rust has a dependency on Tokio, which requires developers not only to implement asynchronous coding patterns but also to handle dependencies carefully.
For easier adoption of this SDK, I’d recommend you to visit this awesome repository(github.com/tidal-music/aws-fastly-http-client) created by Tidal Music, which enables you to have the AWS Rust SDK connect with backend servers through custom http client instantiated from FastlyHttpClient with ease. Simply git clone the project, and add your code and dependencies needed for your project - the final code should look something like below;
#[tokio::main(flavor = "current_thread")]
async fn main() {
let custom_client = lib::FastlyHttpClient::from("origin_0");
let config = aws_sdk_s3::Config::builder()
.region(Some(Region::new("us-east")))
.endpoint_url("https://us-east.object.fastlystorage.app/")
.stalled_stream_protection(StalledStreamProtectionConfig::disabled())
.identity_cache(IdentityCache::no_cache())
.credentials_provider( aws_sdk_s3::config::Credentials::new(access_key, secret_key, None, None, ""))
.http_client(custom_client)
.behavior_version(BehaviorVersion::v2024_03_28())
.force_path_style(true)
.build();
let client = aws_sdk_s3::Client::from_conf(config);
let mut response = client
.list_buckets()
.send()
.await
.expect("unable to list buckets");
for bucket in response.buckets() {
let bucket_name = bucket.name().expect("no bucket name");
println!("bucket_name:{}", bucket_name);
}
Response::from_status(StatusCode::OK).send_to_client();
}
Side note; There's a caveat with this Rust solution - you need to use fastly crate version 0.9.x alongside Rust compiler version 1.79.0. If you upgrade the Fastly crate version, you'll encounter compilation errors (e.g. type mismatch etc), as of writing this.
What's next?
The approach we've demonstrated using these AWS SDKs is just one way to connect Fastly Compute to Fastly Object Storage. The flexibility of S3 client options, combined with the injection pattern of custom http clients, opens up a world of possibilities for leveraging existing code assets and achieving intuitive development at the edge.
I encourage you to take advantage of these SDKs and libraries to enable smooth integration with Fastly Object Storage and other services, and set the stage for your next wave of innovation. Also, don’t forget to visit the forum and stay informed about the latest advancements of Fastly Object Storage. We welcome your contributions and look forward to seeing the innovative solutions you develop!
Top comments (0)