Redis at the Edge with Cloudflare Workers

Computing at the Edge is one of the most exciting capabilities in recent years. CDN allows you to keep your files closer to your users. Edge computing allows you to run your applications closer to your users. This helps developers to build globally distributed, performant applications.

Cloudflare Workers is the leading product in this space right now. It gives you a serverless processing environment without cold starts. You leverage Cloudflare's global network to minimize latency of your applications. You can write your functions in Javascript, Rust, C and C++.

Similar to Serverless functions (AWS Lambda etc.), Cloudflare Workers are stateless. As you can see in Cloudflare’s survey, developers are asking ways to connect their databases from Edge functions. Unfortunately, most databases are not designed for serverless environments, they require persistent connections. We developed the REST API over Redis to enable serverless edge functions to access Upstash in the simplest and fastest way possible.

Comparing with Cloudflare Workers KV

Cloudflare has a basic Key Value store that you can use to store the state of your Edge functions. Upstash Redis excels against Cloudflare KV in several aspects:

• Cloudflare KV provides only basic get/set/delete functionality. Upstash gives you all Redis data structures where you can build much more sophisticated functionality (Hashes, Lists, SortedSets, ranges, appends, increments etc).
• Cloudflare KV is designed to be accessible from the Cloudflare ecosystem while you can access and consume the Upstash Redis from anywhere as it supports both Redis and REST API. You can offload your data from Edge to Redis then process by any Redis client.
• Cloudflare KV is optimized for read heavy applications. The writes can take 60 seconds to be replicated to other locations. The write latencies are in milliseconds for Upstash.

Now let’s write a simple example to showcase the power of Cloudflare+Redis combination.

Analytics at Edge

Web analytics is a very common need for all website owners. Google Analytics is powerful but many developers do not like sharing their traffic data with Google and injecting cookies to users’ browsers. As your website traffic comes through CDN, you can easily track your traffic at the Edge layer. Here we will implement a very simple example to showcase tracking user traffic from Cloudflare Workers. We will intercept the traffic in the Cloudflare Workers and save the user requests to Upstash Redis. Then we will write a basic standalone application which will analyse the requests and give me the following information on a selected day:

• Page views
• Unique visitors
• Top countries with the most number of visitors
• Most visited pages

Setup

If you do not have one, create a database following this guide. In the database page, click the button REST API and copy the REST URL. If you have a Cloudflare account, you can create a Workers function or you can use the playground without an account too.

Cloudflare Workers Code

Cloudflare Workers function accepts request as a parameter. Using the REST API of Upstash, I record the request to a Redis List. I use the current date as the key for the Redis List. So we record the requests per day in separate Lists.

Upstash REST API requires an endpoint and a token. After creating a database, you can copy the endpoint and token from the console clicking to the REST API button. Update your Cloudflare Workers function with the below code, replacing endpoint and token:

const endpoint = 'REPLACE_UPSTASH_REST_ENDPOINT'
const token = 'REPLACE_UPSTASH_REST_TOKEN'

async function recordRequest(request) {
let d = new Date();
let datestr = d.getFullYear() + "-" + (d.getMonth()+1) + "-" + d.getDate();
 let data = [["url", request.url], ...request.headers]
 let url = endpoint + '/lpush/' + datestr
const init = {
   body: JSON.stringify(data),
   method: "POST",
   headers: {
     "Authorization": "Bearer " + token
   },
 }
 return await fetch(url, init);
}

async function handleRequest(request) {
 recordRequest(request);
 return new Response("My Awesome Website"); 
}

addEventListener("fetch", (event) => {
 event.respondWith(
   handleRequest(event.request).catch(
     (err) => new Response(err.stack, { status: 500 })
   )
 );
});

Analytics Tool Code

Now let’s write a simple command line application which will take the date as an argument and return us analytics data. Create a folder and run npm init. Then install the Redis client with npm install ioredis. Copy .env.example as .env file and set your Redis URL (ioredis). Update the index.js with below:

const Redis = require("ioredis");
require('dotenv').config()
console.log("EDGE analytics with CloudFlare Workers and Upstash Redis.")
let redis = new Redis(process.env.REDIS_URL);
let dateArg = process.argv[2]
let datestr;
if (dateArg) {
   datestr = dateArg;
} else {
   let d = new Date();
   datestr = d.getFullYear() + "-" + (d.getMonth()+1) + "-" + d.getDate();
}

redis.lrange(datestr, "0", "-1", function (err, result) {
   let pageview = 0;
   let users = new Set();
   let countries = new Map();
   let paths = new Map();
   if (err) {
       console.log("Could not connect to Redis.")
       console.error(err);
   } else {
       console.log("Upstash Redis Connection Successful. Analyzing the access logs...")
       if (!result || result.length === 0) {
           console.log("Could not find any access logs for the date:" + datestr);
       } else {
           for (const elem of result) {
               let entries = JSON.parse(elem)
               for (let entry of entries) {
                   if (entry[0] === "accept") {
                       if (entry[1].startsWith("text"))
                           pageview++;
                   }
                   if (entry[0] === "cf-ipcountry") {
                       let temp = countries.get(entry[1]);
                       if (!temp) {
                           temp = 0
                       }
                       countries.set(entry[1], temp + 1);
                   }
                   if (entry[0] === "url") {
                       let temp = paths.get(entry[1]);
                       if (!temp) {
                           temp = 0
                       }
                       paths.set(entry[1], temp + 1);
                   }
                   if (entry[0] === "x-real-ip") {
                       users.add(entry[1]);
                   }

               }
           }
           countries = new Map([...countries.entries()].sort((a, b) =>
               b[1] - a[1]
           ));
           paths = new Map([...paths.entries()].sort((a, b) =>
               b[1] - a[1]
           ));
           console.log("\nDATE: " + datestr)
           console.log("\nPAGE VIEWS: " + pageview)
           console.log("\nUNIQUE VISITORS: " + users.size)
           logMap("TOP COUNTRIES", countries);
           logMap("TOP PAGES", paths);
       }
   }
});

function logMap(title, data) {
   console.log("\n" + title);
   console.log("----------------");
   console.log(mapEntriesToString(data))
}

function mapEntriesToString(entries) {
   return Array
       .from(entries, ([k, v]) => `${k} : ${v}\n`)
       .join("");
}

Now you can run your application with node index 2021-6-16 or just node index. The latter one will query for today.

You can think of this example as a starting point. You can develop a rich web application with charts and tables according to your analytics needs. You can use other Redis data structures for more powerful analysis.

Upstash Edge Roadmap

REST API is the first step in our Edge story. We are planning two important developments this year.

• Edge Caching: Right now, all your REST requests come to your database region. We will support Edge Caching soon, so your REST requests will be cached at all edge locations globally. This will enable Upstash Redis to provide low latency all around the world like the Cloudflare KV.
• Global Replicated Databases: Global (multi region) replication will replicate your data to multiple regions. So all requests (both Redis and REST API) will go to the closest location. This will give you low latency globally with the least sacrifice on consistency.

I am planning to write a blog post dedicated to our Edge roadmap. Stay tuned and follow us on twitter.