If you're looking for a simple yet extensible open-source CI platform, you can stop looking now and go to https://drone.io. Drone CI is fairly new platform to power your Continuous Delivery workflow. It's open-source, container-native, extensible with pre-made plugins, and very easy to setup. In fact, one of the things that will get you furious immediately after setup is Docker layer caching.
Caching in Drone isn't handled natively but can be achieved through custom plugins. There are plugins to cache artifacts in AWS S3 (plugins/s3-cache), GCP Cloud Storage (homerovalle/drone-gcs-cache), Host Volume (drillster/drone-volume-cache), and even with External Server via SFTP (appleboy/drone-sftp-cache). The problem arises when you want to build an image and push it to registry using DinD (Docker in Docker). To build and push image in AWS ECR, for instance, you'll use plugins/ecr which uses DinD internally. Since it's inside docker-ception, artifacts created during the build step isn't easily available. What's available are the Docker layers which you can probably cache using the plugins but I don't think is a great idea. Fortunately, there is a better way! Starting with Docker 1.13, docker build command can accept --cache-from argument which specifies a tagged image as cache source. This will pull the image of the specified cache source and only build the layers from bottom until the end of the build when it is different. You might be thinking that this is awesome. It is. But be mindful of some caveats:
Since it will pull the cache source image, make sure it is light in bytes. You might end up furious about the time pulling it.
If you end up changing something that causes the bottom layer to update, specifying a cache source might have no impact.
In addition, you can also change the Storage Driver used by DinD for lesser overhead in I/O. The Storage Driver in Docker manages how the layers read/write files. Best way to choose Storage Driver used by DinD is to match it with the host. You can check via docker info in the host machine. Most probably it will not match the default value for DinD which is vfs. The vfs Storage Driver does some overhead because it deep-copies files from the Docker layers to the writable-layer; but doing so makes it compatible with most environments (e.g. Windows, Linux, Mac). For Linux machines, overlay2 is usually used.
Right now, you might be looking at your .drone.yml file and wondering "how the hell do I specify the cache source image and Storage Driver?". Be less furious. If you are pushing to Docker Hub and is using plugins/docker you can specify the storage_driver setting and then set the environment variable PLUGIN_CACHE_FROM to your cache source image. On the other hand, if you are pushing to AWS ECR or other registry, there might be no storage_driver setting; instead, you have to set PLUGIN_STORAGE_DRIVER environment variable. Drone CI is fairly new and these things aren't documented yet. But it's open-source and checking it is a good practice. :)
The rest of this post shows sample configuration and before-and-after build times. Have fun droning!
Top comments (0)