Connecting your CDN to Fastly Object Storage

Introduction

In this article, I'm going to explain how to get started with Fastly Object Storage and connect it to your Fastly CDN service. In just a few minutes you’ll be up and running with assets stored on Fastly Object Storage and served from your own URL!

First, what is Fastly Object Storage? It’s an S3 compatible object storage with no egress costs when you connect your Fastly Object Storage as an origin to your CDN service.

Fastly Object Storage is always a private bucket, which means you have to provide your CDN service with access and secret keys to connect to it. Therefore, you can’t simply use the object storage URL as the origin. You will need to add a bit of VCL to make it work, but it's not difficult.

Getting Started

To interact with Fastly Object Storage, you will need to create an access key in the Fastly control panel. This key consists of an access key ID and a secret key, which are used to authenticate requests to the S3-compatible API.

1. Log in to the Fastly web interface.
2. Navigate to Resources > Object Storage.
3. Click Create Key.
4. Enter a description and select the scope of access (read or read/write).
5. Be sure to note the access key and secret key, as the secret key will not be visible again.

Pro tip: For better security, you can restrict your API key to only access a specific bucket.

Now that you have created your keys, make sure you keep them somewhere safe. We are going to use them to create our first bucket.

Create your first bucket and upload files

Currently Fastly does not have a UI for creating a bucket and uploading your first file. However, you can accomplish it by using the AWS S3 CLI or any of the numerous packages and libraries available.

To make the process as easy as possible for you, I made a small GUI in Node.js that allows you to interact with the Fastly Object Storage seamlessly.

You can install the app with npm or Docker

Npm :

npx fastly-object-storage-easy-ui

Docker :

docker run -d -p 3009:3009 antoinebr/fastly-object-storage-easy-ui:latest

If the app is installed successfully, you can access it by visiting http://localhost:3009/.

Next, set your credentials in the UI and select the region where you want your files to be stored, along with the access key and secret key. Once that is done you can start creating your first bucket.

After creating your first bucket, click on “View files” to open the bucket, then proceed to your first upload.

How to access your files in the buckets ?

After uploading a file to the bucket, you will not be able to use the bucket URL to access the file directly, such as:

❌ https://eu-central.object.fastlystorage.app/<bucket-name>/<file-name>

You either need to use a proxy app which connects to the S3 bucket via API :

In this case, you can use a proxy app (as shown in the diagram above) and the speed will be limited to ~ 150Mbps and a max 100 req/s per bucket.

Use your bucket with your CDN service

The recommended way to use Object Storage is to connect a Fastly CDN service (via VCL) or Compute to your bucket. By doing so, no egress fees will apply, and there are no limitations with respect to speed or requests per second per bucket.

Integrating with Fastly CDN

Create a new service

You may already have an existing service, but for demonstration purposes, I’m going to create a new service:

The important part at this stage is to set the host. In my case, it is:

eu-central.object.fastlystorage.app

If you created your bucket in another region, set it to:

<your-region>.object.fastlystorage.app

Next, we need to inform our CDN service how to interact with our bucket. To do this, we need to add VCL snippets.

VCL Code for Object Storage Connection

Below is a VCL code snippet for connecting a Fastly CDN service to an Object Storage bucket. This code handles authentication and request signing for S3-compatible API requests.

Copy and paste the code in a vcl_miss (within subroutine).

/*

object storage connection

View Source 

Priority: 100

Type: miss

*/

declare local var.awsAccessKey STRING;

declare local var.awsSecretKey STRING;

declare local var.awsS3Bucket STRING;

declare local var.awsRegion STRING;

declare local var.awsS3Host STRING;

declare local var.canonicalHeaders STRING;

declare local var.signedHeaders STRING;

declare local var.canonicalRequest STRING;

declare local var.canonicalQuery STRING;

declare local var.stringToSign STRING;

declare local var.dateStamp STRING;

declare local var.signature STRING;

declare local var.scope STRING;

# Change these values to your own data

set var.awsAccessKey = "your_access_key"; # 👈 Update this

set var.awsSecretKey = "your_secret_key"; # 👈 Update this

set var.awsS3Bucket = "your_bucket_name"; # 👈 Update this

set var.awsRegion = "eu-central"; # 👈 Update this with your region of choice

set var.awsS3Host = var.awsRegion ".object.fastlystorage.app";

if (req.method == "GET" && !req.backend.is_shield) {

 set bereq.http.x-amz-content-sha256 = digest.hash_sha256("");

 set bereq.http.x-amz-date = strftime({"%Y%m%dT%H%M%SZ"}, now);

 set bereq.http.host = var.awsS3Host;

 set bereq.url = querystring.remove(bereq.url);

 set bereq.url = regsuball(urlencode(urldecode(bereq.url.path)), {"%2F"}, "/");

 set var.dateStamp = strftime({"%Y%m%d"}, now);

 set var.canonicalHeaders = ""

   "host:" bereq.http.host LF

   "x-amz-content-sha256:" bereq.http.x-amz-content-sha256 LF

   "x-amz-date:" bereq.http.x-amz-date LF

 ;

 set var.canonicalQuery = "";

 set var.signedHeaders = "host;x-amz-content-sha256;x-amz-date";

 set var.canonicalRequest = ""

   "GET" LF

   bereq.url.path LF

   var.canonicalQuery LF

   var.canonicalHeaders LF

   var.signedHeaders LF

   digest.hash_sha256("")

 ;

 set var.scope = var.dateStamp "/" var.awsRegion "/s3/aws4_request";

 set var.stringToSign = ""

   "AWS4-HMAC-SHA256" LF

   bereq.http.x-amz-date LF

   var.scope LF

   regsub(digest.hash_sha256(var.canonicalRequest),"^0x", "")

 ;

 set var.signature = digest.awsv4_hmac(

   var.awsSecretKey,

   var.dateStamp,

   var.awsRegion,

   "s3",

   var.stringToSign

 );

 set bereq.http.Authorization = "AWS4-HMAC-SHA256 "

   "Credential=" var.awsAccessKey "/" var.scope ", "

   "SignedHeaders=" var.signedHeaders ", "

   "Signature=" + regsub(var.signature,"^0x", "")

 ;

 unset bereq.http.Accept;

 unset bereq.http.Accept-Language;

 unset bereq.http.User-Agent;

 unset bereq.http.Fastly-Client-IP;

}

In the code, update the following lines:

set var.awsAccessKey = "your_access_key"; # 👈 Update this

set var.awsSecretKey = "your_secret_key"; # 👈 Update this

set var.awsS3Bucket = "your_bucket_name"; # 👈 Update this

set var.awsRegion = "eu-central"; # 👈 Update this with your region of choice

Save your snippet and activate your service.

Access your files :

After activating your service, you should be able to access your files from your CDN service.

https://<domain-name>/<bucket-name>/<file-name>

If you managed to follow the instructions up to this point, you should have a working connection between your CDN and Fastly Object Storage! Congrats 🎉

Extra tips :

To go beyond the basics of connecting your bucket, it's important to optimize delivery and caching performance. Here are a few tips for you:

Enable Segmented Caching for Large Files

Segmented Caching is essential for serving large files efficiently. It allows Fastly to split resources into smaller chunks, reducing load on the origin server and improving performance for Range requests.

Example :

Imagine a video streaming service hosting large videos (e.g., 4 GB movies) on Fastly. A user requests a portion of a video using a Range header to resume playback at 1 GB.

To enable Segmented Caching, add the following code to the vcl_recv subroutine:

# Enable Segmented Caching for large files

if (req.url.ext == "mp4") {

 set req.enable_segmented_caching = true;

 # Consider changing the segment size to reduce volume of requests going back to origin

 # The default is 1 MB. This would generate a 100 requests to origin for 100 MB file

 # or 1000 requests for a 1 GB file. The maximum size is 20 MB or 20971520 bytes

 # set segmented_caching.block_size = 20971520;

}

Enable Streaming Miss

The Streaming Miss feature ensures that responses are streamed back to the client immediately, reducing first-byte latency.

To enable Streaming Miss, add the following code to the vcl_fetch subroutine:

/*
Streaming Miss Feature
View Source 
Priority: 100
Type: fetch
*/

# Streaming Miss does not work with Gzip so you may want to wrap this in a condition
# based on file types that are likely to be Gzipped for example streaming manifests (m3u8 and mpd)
set beresp.do_stream = true;

Join the conversation in our forum

We would like to hear how you are using Fastly Object Storage. Be sure to visit our forum to let us know how we can support you better and to stay up-to-date on new capabilities released for Fastly Object Storage.