I remember times when setting up CI meant long hours of server setup, Jenkins studies, XML configuration pain, and Java memory issues. No wonder continuous testing and integration was not the first thing that came to mind when starting a new project.
Luckily, nowadays things are different. Services like GitHub Actions, CircleCI or Travis (just to name a few) are cloud services that handle the CI flows swiftly for you. Pipeline configurations can be stored neatly next to your code in the version control repository and the changes to them are picked up automatically whenever new commits are introduced. Furthermore, containerization technologies have simplified setting up external dependencies, so even complex end-2-end test environments can be configured only with a couple of lines of YAML.
And the best part is that these services are usually free for open source projects.
However, one thing that has puzzled me over the years when using the aforementioned services is how to avoid DRY a.k.a. Don't Repeat Yourself principle. Many projects have similar CI configurations especially if they are related or implemented using the same technology. In the past, I have found myself copy-pasting configurations between projects or repeating similar steps over and over again and this is something I would rather always avoid.
Let's take an example from the world of Node.js. Unit testing CI process for a node project consists typically of the following steps:
- Select the base OS and OS version for the job
- Select (and install) Node.js version
- Clone the project sources
- (If a cache is found) load cached dependencies
- Install dependencies
- Run tests
- Save dependencies to cache
Steps 1-2 may be combined if you use a system that allows the jobs to be run in a container: the base container can have all the global bells and whistles preinstalled that your project needs. But the rest of the steps you usually cannot avoid defining and actually only one, step number 6, interests me as the application or service developer. After figuring out and defining this process for the first time, I would like to be able to reuse the configuration in multiple projects.
Fortunately, the CI services are constantly introducing new features and most of them have enabled the sharing of common modules in a way or another.
Especially before GitHub Actions was made available for all GitHub users, my standard choice of CI was CircleCI. During last Hacktoberfest I happened to notice that CircleCI had an extra challenge for crafting orbs. I got intrigued and it turned out that the orbs were just the thing I had been missing a few years back: "A reusable package of YAML configuration that condenses repeated pieces of config into a single line of code."
So I decided to give the orbs a go and replaced the unit test job in one of my personal Node.js projects with a job from an orb. The CI configuration was trimmed exactly as I wanted: I got rid of the lines related to cloning, dependencies installation, and cache handling, and all that was left was the test command. The CI configuration was trimmed by 15 lines and everything worked as before the change.
In this case, the common functionality was authored by CircleCI. Node orb that I utilized in my personal project above provides Node.js related basic workflows related to dependency management and testing. But how about creating an own orb? The Hacktoberfest challenge was just about that, i.e. providing some meaningful CI functionality as an orb module that could be used by multiple repositories.
As it happened, I had just come across an interesting documentation tool, alexjs, that lints documentation files for insensitive and inconsiderate writing.
Alex helps to find "gender favouring, polarising, race related, religion inconsiderate, or other unequal phrasing". I thought that Alex would be a good addition to many of my projects, especially as I am a non-native English speaker and take gladly all the help I can get when authoring the project documentation. Also, the use of alex requires the npm toolchain as it is implemented with Node.js. So the additional motivation to implement an orb for alex was to use it in projects based on other technologies than Node.js.
CircleCI hosts a registry for all available orbs so before you can use your self-authored orb in a CI workflow, you need to publish it to the registry. The easiest way to launch your orb development is to install circleci cli and use an orb development kit that sets up a project template for you to get started with the development. Make sure to configure circleci cli first if you try to use the development kit at home.
The orb project template may seem overwhelming at first.
In the source folder there are placeholders for the following sections:
| concept | description |
|---|---|
| commands | steps and their parameters and mapping of those to shell scripts |
| examples | use-case examples, displayed in registry documentation |
| executors | environment in which the steps of a job will be run |
| jobs | full jobs definitions that may combine steps even from other orbs |
| scripts | shell scripts that implement the actual command functionality |
| tests | test scripts for the shell scripts |
For me it took a while to get my head around these concepts in order to achieve the end result I wanted: provide a preconfigured job that would do the code checkout and linting with alex. The target was that the orb user could just import the orb and call the job in the workflow just as with Node.js testing example before.
The other thing that was a bit challenging to get working at first was the orb CI pipeline. The template project is preconfigured to use CircleCI for orb linting and testing. It even packs and publishes the development version of the orb to the registry and executes the integration tests with an actually deployed version of the orb before publishing the final production version. (I'd say nicely designed flow 😊)
CI execution log after successful PR merge and deployment
However, with some documentation browsing, detective work and just trying out things, I finally got the things working as I had planned and as a result I was able to see my orb in the registry. One key thing was to learn how to setup the CircleCI credentials to the organization context so that CI was able to publish the orb on my behalf successfully.
Using the orb is now straightforward:
version: 2.1
orbs:
alexjs-orb: lauravuo/alexjs-orb@0.0.1
workflows:
lint:
jobs:
alexjs-orb/lint
And if all goes well, the job succeeds and logs can be observed from the CI output:
...
CHANGELOG.md: no issues found
README.md: no issues found
src/README.md: no issues found
src/commands/README.md: no issues found
src/examples/README.md: no issues found
src/executors/README.md: no issues found
src/jobs/README.md: no issues found
src/scripts/README.md: no issues found
src/tests/README.md: no issues found
src/tests/test.md: no issues found
CircleCI received exit code 0
The sources for my orb can be found in GitHub in case you want to take a closer look. In fact, currently all orbs published to the orb registry are open source. There is a definite need for private orbs as well, so it is interesting to see what happens in this front in the future.
What about you? Which CI system do you use and does it have similar support for shared functionality?
Cover image by Patrick McManaman on Unsplash
Top comments (3)
Wow, nice job Laura! 💪 I ran into orbs in docker mooc, but didn't bother to check what they were because I was using Github Actions instead of circleCI. Now it all makes sense 😊
Thanks Anniina! Yes GH Actions is an excellent service! I have heared that they have a way of sharing private actions as well, so in that sense they are a bit of ahead of CircleCI at the moment 😊
Yeah I didn't dive very deep with GH actions, but first impressions were good so I think I need to look into it more 😄