Waiting for Docker Compose services in CircleCI - some options

I've used CircleCI a fair bit over the past couple of years - I love how you can run the tests locally before you push - it's a real safety net (although it still doesn't guarantee I didn't do something silly).

Because my builds tend to be Docker Compose stacks, there's usually a bit of service ordering to do - or more precisely, service waiting. You can't install into a database that isn't ready yet.

Jason Wilder's Dockerize has always been the go-to for ensuring that correct order is restored.

My first exposure to this involved including dockerize into a docker image used specifically to build out the tests - it can then be easily called via docker-compose

      - run:
          name: check if mariadb is responding
          command: docker-compose exec test dockerize -wait tcp://mariadb:3306 -timeout 1

This is a great first step, but surely we shouldn't need to add a binary to an image so we can check if a different container is up and running?

In CircleCI, the most common pattern is actually to install it into the CircleCI environment every time

      - run:
          name: Install dockerize
          command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
          environment:
            DOCKERIZE_VERSION: v0.6.1
      - run:
          name: Wait for db
          command: dockerize -wait tcp://HOST:PORT -timeout 1m

This works (why else would hundreds of sites use it?), and indeed, now CircleCI is onto it, and now includes dockerize in their provided base images alongside docker and docker-compose - nifty huh!

      - run:
          name: Wait for db
          command: dockerize -wait tcp://HOST:PORT -timeout 1m

However, both these CircleCI based solutions have one catch when used with docker-compose - trying to get to the database endpoint to check it - because we use setup_remote_docker, it's abstracted away from the container building the images.

In my case, the mariadb exposes its 3306 port to a 32xxx one, but only on the bridge network created by docker-compose. We can still use this to get to the port, but it's complex to achieve, even locally (I've installed dockerize here from homebrew to demo).

toby@pop-os:~/sites/example$ docker-compose port mariadb 3306
0.0.0.0:32781
toby@pop-os:~/sites/example$ docker network inspect example_default | grep Gateway
                    "Gateway": "192.168.144.1"
toby@pop-os:~/sites/example$ dockerize -wait tcp://192.168.144.1:32781 -timeout 1m
2020/05/31 02:43:15 Waiting for: tcp://192.168.144.1:32781
2020/05/31 02:43:15 Connected to tcp://192.168.144.1:32781
2020/05/31 02:43:15 Command finished successfully.

Victory - now all we need to do is artisan code some bash script to retrieve the port, find the network gateway and craft a command to dockerize, make a cup of tea, and sit back in satisfaction (which will probably last until you realise that CircleCI generates a random name for the network it creates...)

But wait - surely there's a better, more Docker-ish way?

There's three key pieces of information to this puzzle:

1. Docker containers can be used to run single purpose applications (d'oh)
2. Those standalone containers can be joined to a named docker-compose network
3. You can control the name of CircleCI's (and docker-compose) default network - using COMPOSE_PROJECT_NAME as an environment variable

So, if we know what the network is called, surely we can join a container to that network, run dockerize (or something else...) inside that network, then dispose of it?

We make sure to add the COMPOSE_PROJECT_NAME env var to the CircleCI config (this has the benefit of naming your stack logically - which also makes it easier to control the stack independently)

  docker-compose:
    docker:
      - image: circleci/php:cli
        user: root
        environment:
          COMPOSE_PROJECT_NAME: circleci
...
      - run:
          name: Check mariadb is responsive with dockerize
          command: docker run --rm --net circleci_default jwilder/dockerize dockerize -wait tcp://mariadb:3306 -timeout 1m

Dockerize is available on dockerhub as an image - so we can run it (and remove afterwards), attach it to the predictably named network, and use the docker-compose service name and internal port.

toby@pop-os:~/sites/example$ docker run --rm --net example_default jwilder/dockerize dockerize -wait tcp://mariadb:3306 -timeout 1m
2020/05/31 03:54:58 Waiting for: tcp://mariadb:3306
2020/05/31 03:54:58 Problem with dial: dial tcp: lookup mariadb on 127.0.0.11:53: no such host. Sleeping 1s
2020/05/31 03:54:59 Problem with dial: dial tcp: lookup mariadb on 127.0.0.11:53: no such host. Sleeping 1s
2020/05/31 03:55:00 Problem with dial: dial tcp: lookup mariadb on 127.0.0.11:53: no such host. Sleeping 1s
2020/05/31 03:55:01 Connected to tcp://mariadb:3306
2020/05/31 03:55:01 Command finished successfully.

There is another image out there that does a similar thing - dokku/wait - their documentation recommends the use of --link to join it to running containers, but given --link is possibly being deprecated, we can use the same --net strategy as before:

toby@pop-os:~/sites/example-$ docker run --net example_default --rm dokku/wait -c mariadb:3306
Waiting for mariadb:3306  .nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
..  up!
Everything is up

Both these packages allow for multiple endpoints to be checked - so for more complex stacks, you can run multiple stages if needed.

I personally really like using the COMPOSE_PROJECT_NAME in my CircleCI projects - given that I like to run them up locally, I also want to be able to control and view them once running - you can use the -p switch to docker-compose to specify a different project to the folder you're currently in - e.g. to clear it all away once you're done!

toby@pop-os:~/sites/example$ docker-compose -p circleci down -v --remove-orphans
Stopping circleci_php_1     ... done
Stopping circleci_mariadb_1 ... done
Removing circleci_php_1     ... done
Removing circleci_mariadb_1 ... done
Removing network circleci_default
Removing volume circleci_files

Hopefully you've found this useful - please let me know if I've missed any obvious options (I've deliberately not veered into log tailing here!)