A few months ago, I thought it would be cool to add click tracking to my newsletter so I could see which articles I shared were the most popular. Full disclosure, I got the idea from Jeremy Daly. In the off-by-none newsletter, he includes the top 5 articles from each issue plus a few honorable mentions based on the number of clicks they received within the first few days of sending it out.
I copy a lot of ideas from Jeremy, he's typically a pretty safe bet when it comes to projects and building what people want. To be honest, he's the one who inspired me to start the newsletter in the first place, and definitely the reason I started my podcast.
So I set off to create click tracking with utm parameters for the web version of my newsletter. As part of the many automations that power my website, I added an event that would look for links in my newsletter content and automatically add utm parameters to them that represented the issue number. The plan was to use Google Analytics to pull the numbers from the web version and combine them with the numbers from the SendGrid API for the email version of the newsletter.
That never happened.
The Google Analytics API feels like it requires a degree in analytics to make any sense. Plus the SendGrid API didn't quite give me what I wanted. I could get at the click data in a meaningful way if I jumped through a few hoops, burned some sage, and hoped for a full moon. It made the project seemingly impossible.
But luckily for me, a few weeks ago Elias Brange wrote an article about a redirect service he wrote with SST. A week after that, Jimmy Dahqlvist published an article about how he took Elias' idea and made it his own. They both explained how to use CloudFront functions and KeyValueStore to create a blazing-fast redirect service that costs almost nothing to run. I loved this idea and thought I might throw my hat into the ring and enhance the idea a little bit further. This sounded like the way to accomplish the click-tracking project I tried and failed at months prior. So I got to it.
My serverless redirect service
My take on the redirect service is intended to serve a unique purpose: click tracking for my newsletter. I don't really care what the shortened versions of the urls are and I must be able to shorten multiple urls at once. Ideally, I wanted something that I could give a list of links to and get back the shortened versions of them without any thought. Just a basic transform, really.
I also wanted a way to automatically track the number of times each one was visited and to be able to associate a bunch of links together - meaning I wanted to know which links belonged to which issue of my newsletter.
To do this, I had a number of moving parts.
- Parse the links out of my newsletter
- Create shortened versions of each link
- Associate the shortened links with the newsletter
- Update the newsletter with the shortened links
Naturally, this felt like a job for Step Functions. I adapted the work that Jimmy did with his redirect service to fit my additional requirements.
This state machine accepts an array of strings and returns an array of objects mapping the shortened url to the provided url. Notice there are no Lambda functions, I was able to get by using only direct integrations and intrinsic functions, which is always a fun goal of mine.
A couple of things to note here:
- I am adding the shortened/original link mapping to both CloudFront KeyValueStore and DynamoDB. More on this in a second.
- Every time you add a value to a KeyValueStore the eTag on it changes. When adding multiple links, I needed to describe the store every iteration to get the current eTag because it is required when adding a value.
- This means I had a parallel count of 1. I had to get the current eTag version before adding a value to the store every time. I couldn't run multiple threads to get through the work faster.
- You can create random strings via intrinsic functions! It's kinda messy, but possible. Check out the state below 👇
{
"partOne.$": "States.Format('{}{}', States.ArrayGetItem($.characters, States.MathRandom(0, States.ArrayLength($.characters))), States.ArrayGetItem($.characters, States.MathRandom(0, States.ArrayLength($.characters))))",
"partTwo.$": "States.Format('{}{}', States.ArrayGetItem($.characters, States.MathRandom(0, States.ArrayLength($.characters))), States.ArrayGetItem($.characters, States.MathRandom(0, States.ArrayLength($.characters))))",
"partThree.$": "States.Format('{}{}', States.ArrayGetItem($.characters, States.MathRandom(0, States.ArrayLength($.characters))), States.ArrayGetItem($.characters, States.MathRandom(0, States.ArrayLength($.characters))))",
}
The value in the $.characters field is an array of the uppercase and lowercase alphabet, plus 0-9. The fields I'm creating are selecting two random characters from that array. In a subsequent state, I string all the couplets together to form a random 6-character string for my redirect link. You might be asking yourself, "Why on Earth is Allen doing it like this?" Great question!
A single field in a Step Function state can have at most 10 intrinsic functions on it. Since each character I'm randomly grabbing takes three intrinsic functions, I can do at most three letters at a time. To be honest, I did the math wrong during the implementation and thought three characters would put me over 10, so I went with two to be safe 🤣
So for each link, I get the current eTag of the KeyValueStore, create a random 6-character string for the redirect, and add the mapping to both the KeyValueStore and DynamoDB.
DynamoDB usage
I needed to include DynamoDB in the solution for a couple of reasons. But first, let's take a look at what the architecture looks like when somebody makes a call to the redirect service.
CloudFront functions don't have access to other AWS services besides KeyValueStore, so I'm unable to update the click count directly from the function itself. Instead, I write a console.log statement when there's a hit on a link to write logs to CloudWatch. A CloudWatch log subscription filter watches for logs in the format of my hit message and triggers a Lambda function asynchronously as a result. This Lambda function does have access to other AWS services like DynamoDB, so I use it to increment a counter for the link stored in the database.
The log event contains the link, which I've used as the partition key in DynamoDB. I update the count property using an update expression and how many times the link appeared in the logs. Here is an example record for a link.
| pk | sk | link | count | campaign (GSI pk) | linkPosition (GSI sk) |
|---|---|---|---|---|---|
| eqXJ8a | link | https://x.com/allenheltondev | 34 | Issue 99 | 22 |
When incrementing the count for a specific link, I can do a lookup via the pk and sk. But if I want to see all the links for a particular issue of my newsletter, I can query the GSI with the issue number and get all the links in the order they are included in the content.
Using the data
All of this would be fruitless if I didn't do anything with the click tracking. I set a goal for myself in 2024 to lift up as many community voices as I can. With that in mind, I decided to add two community spotlights each week with the click-tracking data.
On Thursdays, the Ready, Set, Cloud X account will post a message featuring the most clicked-on social media profile of one of the content creators of the last newsletter issue. On Friday, it will highlight the most clicked-on article or video from the issue. This way we get multiple opportunities to let you all shine!
To do this, I once again went to Step Functions. This time, I kept it much simpler.
This function is triggered via an EventBridge schedule and is provided the campaign to look up as part of the input. The Lambda function will query DynamoDB with the provided campaign and sort the links by click count. Then it pops the top link off the sorted list for both social media profiles and content. To determine if a link is a social media profile, I compared the link with a hardcoded list of sites like X/Twitter, LinkedIn, and GitHub. If the url starts with one of those, it's a profile. If it doesn't, it's content. Simple enough.
To keep the social posts fresh, I wrote 10 template messages for both the profile feature and the content highlight. Step Functions will randomly choose one of the template messages, add in the top link, and use my automatic social post scheduler to schedule and send the messages for me.
Updating the existing newsletter service
Everything I just described is new. It can be standalone as its own thing, as it's completely isolated and triggerable by a set of events. But I needed this to work as a step in my staging workflow for the newsletter. I needed the redirects to be added before staging the email in SendGrid. So I added a few steps to the existing state machine to incorporate the new service.
I added a couple of Lambda functions to parse the links out of the newsletter content and update the source code with the new redirects (reminder - Ready, Set, Cloud is written completely in markdown). Instead of asynchronously invoking the Create Redirects State Machine and using the callback pattern with events to resume the workflow, I thought it would reduce complexity greatly to simply invoke the state machine synchronously. So far, that seems to have been the right call.
The last update I made to the existing workflow was to set up the EventBridge schedule that triggers the social post creation state machine. I set the schedule to run on the next Thursday since the newsletter is published on Mondays, to give my readers some time to click on what they like and get some good numbers. The schedule directly invokes the state machine that gets the top links and provides it with that initial input of which newsletter issue to use as the campaign.
Looking Ahead
So what can you look forward to as a result of all this work? Well, for starters, you get to see which content and content creator really shines in each newsletter. Seeing which content resonates with the community also helps me decide what you all like and how to tailor the content in each issue appropriately.
Be sure to follow the Ready, Set, Cloud account so you can see it in action! Every Thursday and Friday you'll see a message highlighting one of y'all in the community 💙
Beyond the social posts, I'm going to use this for automated sponsor emails. As part of a standard sponsorship, I share with the sponsor how many times their links were clicked to show how effective their ad was. This capability allows me to do that easily and automatically - so that takes work off my plate!
A limitation of using CloudFront KeyValueStore is that it has a max data size of 5MB. While that's a lot of storage for simple redirects, it's not unlimited. Eventually, I'll create another workflow to go back 6 months or a year and replace the redirects back with the original links and remove them from my KeyValueStore to free up space.
I'm sure there will be other clever use cases that come from this project and I'm always looking for ideas and opportunities to learn something new and make Ready, Set, Cloud more automated and fun. But until then, we'll try out this project and start with highlighting our fantastic content creators.
A big thank you goes out to Elias Brange, Jimmy Dahlqvist, and Jeremy Daly for the inspiration behind this project. I learned so much about CloudFront, edge computing, Step Functions, and plenty of other things. If you want to try this out for yourself, let me know!
Happy coding!
Top comments (0)