Stacking Serverless Data Structures for a Pomodoro App on AWS

BuiltWithAmplify #BuiltWithAppSync

Hello friends! Welcome back to another blog post on the Tech Stack Playbook post, your guide to apps, software, and tech (but in a fun way I promise)!

In this post and video linked below, I'm going to discuss the Pomodoro app I built for AWS' AppSync x Amplify Hackathon, as well as some of the key learnings I found along the way building this full-stack application. One of the coolest elements of this project for me, was the data structure manipulations needed to create a pomodoro stream of events chain together, and then the lift of making my local stack example work in the cloud across many users. I certainly learned a lot from this project, so I hope you get a lot out of this blog post and video too! Without further ado, let's dive in 🚀

YouTube Video + Demo:

Application page:

Link: https://prod.d1hzn1hof27la1.amplifyapp.com/

Step 1: Quick Design Mockup (aka map it out)

Something I try to get into the habit of is designing at the most basic level what it is I want to build. It's easy to let code dictate the design, but I think pausing to envision the outcome you want first helps you (1.) arrive at the destination more quickly, and (2.) ensure you are working towards the outcome clearly and efficiently.

This is certainly not a masterpiece, but it helped me start to think about the data structures that would be involved and what I would need:

Step 2: Architect the Tech Stack

This is another important preliminary step that is needed to ensure a smooth outcome for your project or software. When determining the tools you will use, there are about 3 important criteria to consider:
(1.) bandwidth - how much time, how many people, how quickly?
(2.) expertise - how much do I know, how much needed to learn?
(3.) flow - how can this be done easily, efficiently, correctly?

The below are the technologies I used to build this application. You don't necessarily need to use all of them, but I have a strong preference for using Next.js/TypeScript for it's excellent server side rendering (SSR) functionalities, lightning fast build/render time, and it's integration with Styled Components for building customized front-ends. For AWS, the AppSync, DynamoDB, Lambda, S3, Amplify stack is one I am particularly partial for. By using the Amplify CLI, we can coordinate all of our AWS resources for our CloudFormation stack and make sure everything is packaged up cleanly before we send to the cloud and eventually our users.

Here is a list of the technologies I used to build this app:

📲 Front-End:

• Next.js: modern app development with Server Side Rendering, dynamic routing, and more
• TypeScript: to enforce type safety throughout functions and API requests
• Styled Components: for fast component development and scaling UIs
• Material UI: React component library

⚙️ Back-End:

• 🚦 API: GraphQL <> AWS AppSync: managed API service and GraphQL transformer based on written schema
• 🗃️ DATABASE: Amazon DynamoDB: lightning fast NoSQL database optimized for web apps and fast READS/WRITES
• 💻 HOSTING: Amplify Hosting: managed service optimized for Next.js v13's features and deployment requirements
• 💾 STORAGE: Amazon S3 highly inexpensive and highly scalable storage solution for images and app hosting
• 🔐 AUTH: Amazon Cognito: managed authentication service that neatly wraps into fine-grained-access-controls around the API

Step 3: Figuring out numeric sort querying using GraphQL of NoSQL DynamoDB data

This was probably the hardest step for me in this project because at the highest level, when you are running a data stream with chaining, you affect one item and it means you must alter the other elements in the chain accordingly.

TL;DR: you edit one item, so you must edit the others accordingly.

For example, consider the following where we will move an item from the 3rd position at index[2] in the stack to the 1st position at index[0] to re-order the stack and everything pushes down.

ORIGINAL     —>   FINAL
[ITEM#1]          [ITEM#3]
[ITEM#2]          [ITEM#1]
[ITEM#3]          [ITEM#2]
[ITEM#4]          [ITEM#4]

This made perfect sense on paper, but the implementation of this as stack manipulation got even more confusing when I tried to model an abridged NoSQL data structure to emulate how DynamoDB would be interpreting this.

I would be using AppSync to query the Pomodoro items so that I could do a LIST of those items for each user based on the order variable. I figured it would be wise to create a separate order integer value in my schema so that I could track a value that would be mutating regularly as the user drag-and-dropped their tasks in their READY stack.

When new items get created, their order would increment by +1 relative to the last item index's order. However, as items get moved around, their order will be updated while the ID of course stays the same. Something like the below was what I needed to solve for:

      ORIGINAL         —>             FINAL
[id: 1111; order: 1]          [id: 3333; order: 1] 
[id: 2222; order: 2]          [id: 1111; order: 2] 
[id: 3333; order: 3]          [id: 2222; order: 3]
[id: 4444; order: 4]          [id: 4444; order: 4]

In the above example, we have unique a id for each project, but the order is changing as id:3333 goes from order:3 to order:1 meanwhile the other 3 items in the stack all have updated order values as well.

This was the big data structures and algorithms conundrum I had to figure out, specifically how AppSync could process these batch updates at once to update many items in DynamoDB at once.

The solution:

I tested out a number of different implementations and ended up going with a Global Secondary Index (GSI) on a value orderList that takes in a String value and has as its sortKeyField order as a integer value.

...
id: ID!
title: String!
order: Int
  // ❌ This GSI didn't work
  @index(
    name: "ItemsByUserID"
    queryField: "listItemsByOrder"
    sortKeyFields: ["timestamp"]
  )
orderList: String @default(value: "CURRENT")
  // ✅ This GSI worked!
  @index(
      name: "OrderItemsByUserID"
      queryField: "listOrderItemsByUserID"
      sortKeyFields: ["order"]
  )
...

I found that if all of the DynamoDB objects maintain this structure of an order number value and an orderList string value, I could do a list query that I named listOrderItemsByUserID where I find all of the items that have orderList === "CURRENT" and then sort it in ascending order, this then allowed me to list the items in incrementing order numerically even though I was doing a query off of a string value.

This query looks like this:

query MyQuery {
  listOrderItemsByUserID(
    orderList: "CURRENT", 
    sortDirection: ASC
    ) {
    items {
      id
      order
      title
    }
  }
}

Step 4: Stacking/Re-Stacking the Pomodoro Data Structure

Once I figured out that we could use that type of query pattern to get the data listing correctly with the numeric sort, I needed to figure out how to serve order changes from the client to AWS.

I was using a JSON test data structure, which was considerably easier to test and troubleshoot with, but I was not familiar with how to do batch-writing to DynamoDB where mutating one item in the stack spikes mutations of all the other items in the stack and then this is automatically served to the cloud and saved in real time. I was a bit skeptical about this type of set-up given I had realized I've mainly worked on saving items individually or in a queue, rather than a for loop to to batch through client-side updates.

This is why this project turned into a very rich learning lesson because I was able to find out a way that you can use AppSync to serve updates to DynamoDB that almost feels like a websocket but significantly less bandwith since we are just doing update mutations rather than instantiating a whole websocket to serve the updates.

I learned about this TypeScript method for batch iterating with these components:

• (1.) A for...of loop: to iterate through the items in the array. We want to run a function across many different items for AppSync to communicate to DynamoDB.
• (2.) The .entries() method: ** for spotlighting the various items in the stack. We call this method over the reorderedItems array so that we can use this method to get back an IterableIterator of [index, element] for all of the items in the stack, which in our data model is [number, ItemDDB].
• (3.) Tuple Desctructuring: We need to assign new values to the [index, element] base structure of the IterableIterator into variables, so this is a method that will let us make and track those changes.

async function updateItemsOrder(reorderedItems: ItemDDB[]) {
        for (const [index, item] of reorderedItems.entries()) {
          try {
            const input = {
              id: item.id,
              order: index + 1,
            };
            await API.graphql(graphqlOperation(updatePomodoroItem, { input }));
          } catch (error) {
            console.error('Error updating item order:', error);
          }
        }
      }

⚠️ Important note with TypeScript: This IterableIterator is an advanced iteration feature specific to ES6 (ECMAScript 2015), and not usable or callable in previous ES versions. TS will default to using ES5, which then means that it won't support the iteration we want to use, so we need to do one of two things:

• 1.) Switch to ES6 in our tsconfig.json file to change "target": "es2015", under the compilerOptions object, or
• 2.) Add the --downlevelIteration compiler flag by adding "downlevelIteration": true, to the compilerOptions object.

Step 5: Hard-Typing with GraphQL and TypeScript

There are numerous reasons for using TypeScript in an enterprise applications, from ensuring error-free and bug-free build code when it goes to production, as well as a consistent and reliable framework for how your data structures should be used or get called in your application.

For this hackathon project, I wanted to practice typing my functions, data structures, and API calls. It was either going to be a great learning experience, or a horrible decision that I would regret forever. Even though this was a hackathon project, we must either go big or go home, as they say.

Type Guards

When we are working with AppSync, we will be making requests to get either one item (GET), or many items (LIST), so we will need to ensure we have an interface that can cross-check the data object and make sure what you expect to get is really what you get in your call.

An example of this is my getItemsFromDDB async function which queries my global secondary index for my schema, passing in the orderList string variable, a request to get the data in ascending (ASC) order, and a filter around the status, as I only want the items that are non-COMPLETED, aka "RUNNING" in my case.

Then I want to make sure that I check against an interface, which I named isGraphQLResultForGetItemsFromDDB which accepts the fetchedItems I receive and make sure that it matches up with this structure:

interface ItemDDB {
  id: string;
  title: string;
  status: string;
  timeBlock: string;
  minutesLeft: string;
  color: string;
  colorDefault: string;
  itemStatus: string;
  order: number;
}

function isGraphQLResultForGetItemsFromDDB(fetchedItems: any
  ): fetchedItems is GraphQLResult<{ listOrderItemsByUserID: { items: ItemDDB[] } }> {
    return fetchedItems.data && fetchedItems.data.listOrderItemsByUserID && fetchedItems.data.listOrderItemsByUserID.items;
}

The whole function, useEffect hook, and type guards looks like this:

  const getItemsFromDDB = async () => {
    try {
      const fetchedItems = await API.graphql({
        variables: {
          orderList: "CURRENT", 
          sortDirection: "ASC",
          filter: {
            status: {
              eq: "RUNNING"
          }}
        },
        query: listOrderItemsByUserID,
        authMode: "AMAZON_COGNITO_USER_POOLS"
      })

      if (!isGraphQLResultForGetItemsFromDDB(fetchedItems)) {
        throw new Error('Unexpected response from API');
      }

      if (!fetchedItems.data) {
        throw new Error('No data found');
      }

      const returnedItems = fetchedItems.data.listOrderItemsByUserID.items;
      console.log(returnedItems);
      setItems(returnedItems);

    } catch (error) {
        console.log('Error running getItemsFromDDB:', error)
    }
  }

  useEffect(() => {
    getItemsFromDDB();
  }, [])

Step 6: Shipping Your Project Live

Unsurprisingly enough, Amplify's managed hosting prevents you from pushing error-ridden code into production (I guess this is for better and not worse). What is good about a thorough review like this is that AWS will not push or update just-pushed changes if there are type errors or anything that causes breaks in the code deployment pipeline.

I thinks this is a great benefit to make sure that you do not push buggy or error-prone code to your users at any given time. This is a great benefit for a clean CI/CD pipeline.

Also, Amplify's way that it seamlessly integrates Server Side Rendering out of the box for deploys is phenomenal.

Further, as you work on your AppSync schema, you can configure and push your code for the back-end outside of the front-end. This decoupling allows you to start branching out beyond a monorepo and start turning Lambdas (this project has a couple lambda functions updating the public data table based on lambda triggers), Authentication, Databases, and storage into microservices that can be worked on and streamed to the front-end via APIs, Amplify's routing service, and AWS service identifiers in AWS Amplify's aws-exports.js file with an authorized IAM role.

Did you learn something new about AWS and the cloud? 💭

Let me know in the comments below! ⬇️

Subscribe to the Tech Stack Playbook on YouTube:

Let me know if you found this post helpful! And if you haven't yet, make sure to check out these free resources below:

• Sign up for my newsletter: Email List
• Follow my Instagram for more: @BrianHHough
• Watch my latest YouTube video for more
• Listen to my Podcast on Apple Podcasts and Spotify

Comments

[tippi-fifestarr]

Love this, is the code closed-source? I signed up (what terms and conditions did I agree to lol?!) and tested it out, works really well. What are your long term plans for this? I saw you did it in a hackathon :D

[Johannes Koch]

Can you add a high level and a detailed architecture diagram for this one :-)?

How does your CI/CD pipeline look for this one? ;)

[Brian H. Hough]

Hey @lockhead! 👋 Thanks for checking out my blog!! 😊

So I have branch-based deploy pattern via GitHub with 2 branches:

• main —> I work on this branch and make PRs up to prod
• prod —> These builds run automatically from an authorized PR from main

The prod branch then goes through a 4-phased deployment process through Amplify hosting:

• provision —> confirm Amplify build setup and project configuration
• build —> take commited codebase, clone-it, then build back-end, and then build front-end
• test —> testing step for preBuild or build tests
• deploy —> ensure continuous uptime of the website and switch in the new version and switch out old

Here is a solution architecture from one of my past projects, but the CI/CD pattern is the same as this one!

It's also included on this tweet:
twitter.com/BrianHHough/status/162...