First impression and takeaways from using Docker to boost local development environments and CI pipelines.

It was November 2016 when a team of four Kainos developers kicked off work at the Civil Money Claims project under HMCTS (MoJ) Reform programme. The name “Civil Money Claims” may not give away all the details by itself, so to provide you with a brief explanation — it’s about giving citizens an easy and convenient way to claim financial compensation online, e.g. for sloppy house repairs work or an unpaid invoice.

The problem to solve introduced a number of integration points we had to cover — payments service, fees registry and an in-house IDAM solution. Adding a bunch of applications we’d create ourselves, it was immediately clear that we needed a tool that would allow us to conveniently spin up environments and orchestrate all those services.

We unanimously decided to give Docker a go on this. Although we were mostly new to the tool, our project was just starting alpha phase and we could always fall back to proven solution in case the experiment turned out to be a disappointment. A modern day software engineer shouldn’t let go an opportunity to get to grips with a new technology, even more so considering how much noise Docker has been making in the IT world over the past few years. 

I can now say that we’re happy with what Docker has given us so far, but that wasn’t without trial and error. I would like to share some takeaways we’ve accumulated over the past few months working on CMC. In fact, at the time I’m writing this our repositories are being open-sourced to a public GitHub organisation — feel free to go through the code as you read along.

Setting up a workflow

Since I wouldn’t like this article to become yet another paraphrase of introductory tutorials, I’ll continue with an assumption that you’ve already had some hands on experience with Docker or know what it is and how it works.

Although both building and running of containers can be done using sole Docker, we’ve found Docker Compose much more convenient to work with. Compose is a tool that’s intended for multi-container setups, but it adds enough value on it’s own that it’s worth using it even for services which use a single container. This is because compose enables you to encapsulate all the logic on how to build the images and run the containers. You provide an appropriate Dockerfile and docker-compose.yml for each of your services and then just use the same set of commands to build or start any of them:

$ docker-compose build

to build and

$ docker-compose up

to run, you don’t need to remember any image names or container IDs. It also behaves a little nicer with regard to disk space — if you stop a service using docker-compose down, it will delete the container once it exits, lifting the burden of having to periodically cleanup stale containers. You still need to keep track of the images yourself though.

A setup that’s been working quite well for us is to provide each project with Dockerfiles containing minimal configuration that’s required for a given service to run properly. Such Dockerfile builds an image that can theoretically be shipped for anybody to use. Then we add a compose file to that which defines which images to build and provides a uniform interface for building the project. Our Claim Store service can be used as an example. We build two images there, one for the service itself and one for the database it requires to run. Both container images will be built when docker-compose build command is issued. This is executed automatically on our CI server and images are published to Artifactory on each successful build.

We then have a master docker-compose.yml file in a separate repository which defines all services that comprise our system. Master compose file does not know how to build the images for each of the services. It instead depends on those images being available in the system’s local cache. There are two ways a container image can land there — either by executing the build locally as it was described above, or by executing docker-compose pull. Pull will instruct Docker to lookup the images in the registry it has configured (the aforementioned Artifactory in our case) and download the latest versions.

There is no magic to this workflow — it’s a standard, proven way of developer collaboration. Code can be shared easily and each developer can work independently of others if needed. It shows that Docker’s design has been given some thought and it aims to conform to industry standards, which makes the adoption more straightforward.

Managing compose files

Continuing on the topic of compose files, we’ve used a master file to both setup a local development environment and execute end-to-end tests of our application. The tests had to execute both on local environments and during CI builds. This probably sounds to you like a number of different use cases and it indeed caused us some headaches before we sorted out a convenient way to handle them all.

Let’s start with local environment. Defining all the services required by the system in the compose file to be able to start them up is a good first step, but it isn’t enough to enable local development. This is because Docker will not expose any ports to the host out of the box, so all the applications will be isolated within Docker and you wouldn’t be able to interact with them easily. Additionally, file system data will by default live for only as long as the container that manages it is running. That means that all your databases will be wiped each time you restart the database container. Does not sound too appealing, does it?

Of course both problems can be remedied easily — Docker provides a mechanism to forward container ports to host ports, enabling interaction with applications that you have running. You can also persist your data between restarts using the volumes feature of Docker, which maps selected container directories to the ones on your local file system. They will not be deleted unless you ask for it explicitly. Both features can be enabled either by passing command line arguments to docker command, or by defining additional ports and volumes entries in the compose file and letting docker-compose do all the work.

Having done that, we had our local environment covered. Developers could do their work without any problems and QA could execute end-to-end tests, clearing persistent data as needed. All done, everybody happy? Alas, not yet!

The compose file configured for local environment wasn’t fitted too well for CI builds. Firstly, we wanted to always have a clean test environment when executing tests on CI and that meant we couldn’t persist data between runs. It’s possible to delete volumes when tests finish and services are taken down, but it’s just unnecessary maintenance overhead. Secondly, we couldn’t have the container/host ports mapping feature turned on — what if two builds were triggered in parallel on the same executor? Or the executor host itself had some of those ports allocated? A conflict would occur and the job would fail.

Our first approach to resolve this problem was to use two master compose files. They mirrored each other, difference between them being that one had features needed for local development added. You can imagine how quickly this became a weight that was dragging development down — every change to services configuration had to be done in two places. Of course we would forget to update one of them from time to time, what lead to time wasted on debugging problems that shouldn’t be there in the first place.

A different approach had to be used. Help came with a Docker Compose feature of overriding services configuration by providing multiple compose files.

Master docker-compose.yml file went back to defining services, their environments and dependencies only and a new docker-compose.local.yml file became a place to set features needed for local environments. This allowed us to cleanly split the two concerns. CI could check out the now plain master file and have a clean, isolated environment available for every test run, while developers didn’t miss out on any features they needed.

Configuring services within Docker containers

Obviously the applications we were developing required some sort of external configuration in order to acquire environment specific values and various secrets needed to connect to databases and other services.

A common way of doing this is to provide a configuration file, but this approach does not match up too well with Docker. The configuration file needs to be available on container’s file system, which you can achieve in two ways. The first one is to embed the file at the time of building an image, but that means your images would no longer be “build once, run anywhere”, but “build once, run on that one specific environment”. Definitely not something you would like to do. The other way is to define a volume and make a directory with the configuration file available to the container. This is an improvement, but making sure that a correct file is available in a specific location is a little too much maintenance in my opinion — don’t forget you also need to cleanup the volume after the container is shut down.

A solution we’ve found to be most suitable was to follow a Twelve-Factor App approach and configure each service using environment variables only.

Docker Compose supplies an environment configuration entry which allows to do this conveniently. You can either hardcode the values that will be passed to a particular service, or make compose pull them from the shell it runs on. Docker’s inspect command can be used if you later need to check which values the the variables resolved to:

$ docker inspect --format '{{ .Config.Env }}' f50637091876

[ROOT_APPENDER=CONSOLE REFORM_ENVIRONMENT=local ROOT_LOGGING_LEVEL=INFO REFORM_SERVICE_NAME=claim-store REFORM_TEAM=cmc JSON_CONSOLE_PRETTY_PRINT=true PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C.UTF-8 JAVA_HOME=/docker-java-home/jre JAVA_VERSION=8u141 JAVA_DEBIAN_VERSION=8u141-b15-1~deb9u1 CA_CERTIFICATES_JAVA_VERSION=20170531+nmu1]

This way you don’t need to care about placing configuration entries in a particular location as it’s done with files. You just need to make sure they are set, which is probably a least work way of feeding config into an application.

Dockerized tests setup

Docker turned out to be excellent when it comes to executing integration/end-to-end tests on a CI pipeline. The principle is quite simple — you startup a Dockerized environment, run tests and take the environment down. All done, proceed to the next CI stage. However, there are a couple of caveats to know about before everything can run smoothly.

In previous sections I’ve mentioned that test environment that is used by CI to execute end-to-end tests is isolated — it’s not accessible outside Docker. Because of this tests cannot be executed from the pipeline directly, they need to be embedded in a Docker container as well. During my early work with Docker, where I’ve used it mostly to startup long running processes such as web APIs and databases, I found it easy to forget that containers can be used to execute any kind of process, including the ones that just do some work and finish in a finite amount of time.

Behind the hood creating such an image boils down to just copying the source files of the test suite and setting a command to run them as an entry point of the container. If you weren’t using Docker you would check out those sources and issue the command yourself. Here it’s almost the same, it’s just that Docker does it on its own.

Tests image is built as a part of the end-to-end tests project CI pipeline and published to registry, making it available just like any other container image. Since it’s a part of the master compose file, each time a pipeline issues a docker-compose pull it will be downloaded alongside all service images.

Another thing that should be trivial but isn’t when using Docker to run tests is determining whether the execution succeeded and acquiring generated output. For the former matter, it’s quite popular to check the exit code of a command that was used to start the tests. Unfortunately, this will not work here because you would be looking at the exit code of docker-compose up or docker-compose start, which will almost always be 0 — the container started up successfully after all. What happened inside it is a different matter.

Luckily, fetching such information is nothing extraordinary when working with Docker. The inspect command which I’ve shown before can help us out again:

$ docker inspect --format '{{ .State.ExitCode }}' 94e8d6c13d8f

0

Docker Compose does not have it’s own equivalent of inspect command, so we need to acquire container’s ID first:

$ docker-compose ps -q end-to-end-tests

94e8d6c13d8f

The end-to-end-tests in snippet above is a name we’ve given to the test suite “service” in the compose file.

As for the issue of interrogating test output produced during the run, some of you will probably have guessed by now — we need to define a volume on test suite container, mapping a container directory to a CI executor’s file system. Results can then be archived/pushed for analysis from there. This approach is assuming that the output is written to files — if for some reason you are forced to use stdout for this, you can easily acquire it as well using docker-compose logs.

The whole concept of running tests in this way is quite straightforward once you become more familiar with Docker. I think it’s also more convenient than having an actual environment that you execute integration tests against, as you don’t need to care about cleaning the databases up after each run and worrying that parallel pipelines will interfere with each other.

Handling dependencies

It’s natural that a system which is decoupled to a number of modules or services will develop a graph of dependencies between those elements. I’ve said in the beginning that those started cropping up for us fairly quickly — we had a number of services delivered by other teams we had to integrate with and our own application was being developed as a distributed system as well.

An additional problem that often comes up in such cases is the ordering — e.g. databases need to be up and running before applications that rely on them can be launched. Docker Compose defines a depends_on service configuration entry that allows to define some ordering between your services. Out of the box, this feature will not work as you would expect it to. Remember the previous section where I’ve mentioned that Docker cares only if a container got kicked off successfully? That’s correct, your database dependency will be treated as satisfied even though PostgreSQL server is only starting up.

At first we’ve managed to get around this problem by using slightly hacky shell scripts which attempted to establish a connection with database server in a sleep/retry loop. Such script would be configured as an entry point of a container that depended on a database and do the polling before starting an actual application. That worked, but the whole thing felt more painful than it should be.

Another thing was determining whether the whole system finished starting up and is functional, especially during CI builds. Our first approach was very naive and involved sleeping for a predefined amount of time and crossing your fingers in hope that everything will be up when the pipeline proceeds to run tests.

That worked while the amount of dependencies was small. An improvement was made after we added health check endpoints to our services. We could then add another polling script which delayed the test run until all services responded with a “healthy” response.

Health checks…

We were quite late to find out that Docker supports them as well and that appears to be the best way of handling dependencies between containers. This can be done either through a HEALTHCHECK directive in a Dockerfile or a healthcheck service configuration entry in docker-compose.yml. The idea is to specify a shell command which Docker will use to deem a container as healthy. The verification is done based on the command’s exit code, 0 being a success as usual. Here’s an example based on a PostgreSQL database container:

HEALTHCHECK --interval=10s --timeout=10s --retries=10 CMD psql -c 'select 1' -d backend-db -U $BACKEND_DB_USERNAME

You’ll need to update your compose file with a condition configuration entry in order to use it, as compose does not verify the health check by default, e.g.:

services:
  ...
  backend-api:
    depends_on:
      backend-db:
        condition: service_healthy
  backend-db:
    ...

The default used by Docker is service_started, which only checks that the container has been kicked off successfully.

Defining a health check on a container will add an additional healthy/unhealthy status to it (docker-compose ps does not display it unfortunately, you’ll see it only from docker ps). You can see it’s possible to provide polling configuration as well. It tells Docker when it should give up retrying and finally treat given container as unhealthy and fail the system startup. There is no rule of thumb for setting those — you’ll need to do a number of empirical tests to see how much time your containers need to startup and then choose values that will allow CI builds to run reliably. Local environment is less of a problem as you can just hit docker-compose up once more if a timeout occurs.

We believe that the health check should be a part of an image definition which you provide in a Dockerfile as a best practice. It makes your deliverable more robust and is a courtesy towards people who will be using it. Defaults you supplied can always be overridden using a compose file if anyone needs to do so.

Space, time, and being prepared for the unexpected

The way I’ve been describing our time working with Docker so far may give an impression that it was all fun and games — a team of developers working out the best way to harness a new, hot and trending technology. The reality is that we did run into some bugs with the software.

The first one was a time drift problem on Macs. If you had any containers running on your local environment and put your Mac into sleep, the clocks of these containers’ operating systems would not resynchronise after waking up and resume from the time your system was suspended. This emerged fairly quickly for us in a form of session cookie age verification problems, but was quite a surprise nevertheless. It was a known bug at the time we hit it and though updating the time on containers was quite easy to do, the problem was still annoying. Luckily the issue is fixed now.

Going further on the Mac problems train, another one was related to disc space. Or, to be more precise, disc space being gradually taken away from you. Docker for Mac maintains a Docker.qcow2 file under ~/Library/Containers/com.docker.docker, which it uses for under the hood work on writing and updating container images. The thing with it is that it does not shrink when containers or images are deleted, and in addition to that is appears to be slowly growing by itself for no apparent reason. To give you an idea of the scale — I’ve recently wiped this file after around six months of working with Docker from a clean system installation and that freed up 56 Gigabytes on my disc. It’s not a critical matter (unless you have a small hard drive), but it is irritating considering that you need to keep track of it and clean up yourself. Both issues are still open on Docker for Mac’s GitHub repo.

Another thing that may be slightly worrying is the attitude of Docker’s development team. They appear to be often changing their vision of Docker, deprecating features that were well received by the users and leaving backward compatibility in question. The health checks and container dependencies feature I’ve talked about before can be used as an example. In this Github question they say that support for depending on container health checks will be dropped in new versions, with the the intention of completely scrapping cross container dependencies (and eventually Docker Compose as a whole) going forward. Suggested solution they want to implement is to have containers that depend on something keep crashing and restarting until their dependency is finally met…

I’m not sure now, maybe this is a way to go. Services should be written with an assumption that anything can crash at any time to be considered robust after all. Still, forcing an application to crash because you don’t want to support health checks feels like a lazy man’s way out. Approach shifts seem to be a little radical too, what reduces confidence you can have in the technology, especially if you were planning to use it in production. We did consider that for a while but decided not to commit in the end. You can find stories of people who did and ended up tearing their hair out.

Don’t get me wrong — I still think that Docker is an awesome piece of software and we probably would have had a harder time without it. We just shouldn’t forget that it’s only 4 years old and still in a state of flux.

Swimming forward

The next thing we have in mind is to do some fine tuning of containers’ memory and CPU allocations. Startup time for our environment has been steadily increasing as services were being added. Now it’s around 2 minutes to get all 15 containers up, which is not bad, but it would always be nice to shave a couple seconds off that. Test suite execution time could possibly be improved as well then.

If you have a non mission critical environment and a use case that Docker could cover I would definitely recommend giving it a chance. It’s a real tool that can empower your development team and save you a considerable amount of time. It’s also a step toward improving your ops skills and getting familiar with containers in general which I feel will become more ubiquitous as the time passes.

Thanks for reading — I hope you’ve found at least some of it helpful and applicable to your own work!

This blog entry was originally published on Medium on 3rd Oct 2017.