Building a Ruby Serverless CI & CD pipeline

Originally published at Codegram's blog.

When facing a serverless project, the last thing we probably want to worry about as developers is infrastructure or deployment. We just want to make sure that as soon as we are certain our app is working OK, it is deployed as easily as possible. This could also be applied to any kind of project of course, but I think it fits specially well in a serverless one.

In this example, we're going to use the Serverless framework, which already does a lot of the work, but I still found myself repeating the same tasks and code between different projects each time I had to create the continuous integration & deployment pipeline.

The pipeline

So, how should this pipeline look? Which steps do we want to have to make sure everything works OK? To me, it should look something like this:

1. The developer works on a new feature on a new Git branch.
2. They create a new Pull Request to keep pushing the changes.
3. A Continuous Integration system runs the test suite and maybe some linters.
4. When the Pull Request is approved and reviewed by their peers, it gets merged to the main branch.
5. The Continuous Deployment system kicks in (which can be the same as the CI) and deploys the new features.

After repeating this process in different projects, I thought it was time to abstract some of the common code and best practices like dealing with dependencies, CI cache, building, deployment and so on.

A real world example

Continuing from a previous post on how to create a Serverless GraphQL API with Ruby, I updated the project to add some tests and auto-deployment with CircleCI. Check out this pull request to see how easy it was!

To start, we'll need a CircleCI account and setup the project with it. Then add the AWS credentials to Circle so it can be automatically deployed. I recommend following these instructions to create an IAM user for your project.

To add the AWS credentials go to the project's settings page, and add the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables with the values you goot from the AWS console. Since this are the default environment variable names, Serverless will use them when deploying.

After that, we need to add CircleCI's configuration to the project:

.circleci/config.yml

version: 2.1

  orbs:
   serverless-ruby: codegram/serverless-ruby@0.0.3

  workflows:
   main:
     jobs:
       - serverless-ruby/test
       - serverless-ruby/deploy:
           requires:
             - serverless-ruby/test
           filters:
             branches:
               only: master

This file uses our Circle CI Serverless Ruby orb (more on this later) and then sets a workflow with two jobs: one to run the test suite and one to deploy the code when it's merged to the master branch.

Finally, we need to tune the serverless.yml file in order to get smaller builds with just these lines:

serverless.yml

package:
  exclude:
    - '**'
  include:
    - app/**
    - vendor/**
    - '*.rb'

These lines tell the Serverless framework to exclude all the files in the root directory and only include our application code at app, the packaged gems at vendor and all Ruby files at the root, since there's where app.rb is located (the project's entrypoint; the file AWS Lambda calls to process each request).

That's it, we now have a fully CI & CD serverless pipeline 🎉.

CI & CD process explained

Early on I mentioned that we were using a CircleCI orb. For those who don't know: orbs are a way to share common configuration, jobs, commands, etc. in a CircleCI configuration file. This way you don't have to repeat the same code and setup between projects and you can benefit when an orb is updated. We created it not only to avoid repeating code but also to share some best practices and solve common issues in this kind of projects.

In the example project, previously to using the orb, the deploy process was far from ideal. Lambda needs to package all your code and dependencies into a single ZIP file and since we're in the Ruby land this means packaging also our gems. Usually when using Bundler all your gem sources are stored in your home directory, but this doesn't work when you need to include them in the ZIP file. So what I needed a Serverless plugin to hook into the package step, run bundle install --deployment to include all the gems content at the project folder, then deploy and then cleanup the mess by removing the vendor and .bundle directories from the project root.

This approach was quite prone to errors since I was deploying from my own computer and the gem dependencies weren't built in the same environment they would run.

So, what does the orb do?

• Builds the gem dependencies using Docker, this allows having gems with native extensions since they'll be compiled with the same environment as AWS Lambda.
• Efficiently manages all the caches so the builds are fast between runs and you don't lose time waiting on them.
• If your project uses DynamoDB, it will install the dependencies and start a database before the test suite runs.
• Runs the test suite with rspec and uploads the test results using RspecJunitFormatter
• Deploys the project with serverless and excludes all test and development dependencies with each commit to master.

All these steps could be migrated to your favourite CI system, as they don't depend on CircleCI.

I'm very happy with the end result, now our Serverless Ruby projects are deployed blazingly fast and with the certainty that everything is working OK. An improvement could be creating different deployments for each Pull Request so we don't have to deploy development environments from our computer, but this is a story for another time!