How to keep the developer focused on developing
Blog post 1 of 3 (maybe) about KTH:s deployment pipeline, built around Docker.
Sometimes you come across good stuff that inspire you to do things right. Three years ago i listened to Jeff Humble talking about Continuous Delivery on the excellent podcast Software Engineering Radio. The show is now three years old, but that podcast had a deep impact on the way KTH has designed it’s deployment pipeline.
Some key features:
- Everything is version controlled, applications, configuration changes and secrets.
- Deploys continuously by Sematic versioning or explicit versions.
- Developers know and like Git (we got inspired by Heroku).
- Use open standards, minimize usage of tools to learn.
Everything is version controlled, applications, configuration changes and secrets.
Versioning has been a key part of industry since the beginning of time. Why? Because it is easy for humans to understand and communicate. But in a continuous delivery world, versions does not really make the cut, software is installed without human interaction. So we want to know exactly what we are running, but also be able to understand what we are deploying. In a “You build it, you run it” environment it is important to be sure exactly which code is running, and how that changed since the last deployment. It is also important to track feature changes back through the development process, maybe all the way back to the original feature request ticket.
Solution: All our Docker images are tagged with semantic versioning and the Git commit hash (in this example bellow c09f25a). The result is that the deployment process can install the latest version but we also know exactly which code is running. No more reading release notes and hope they are correct for that version.
places-api_api replicated 1/1 kthregistryv2.sys.kth.se/places-api:0.1.8_c09f25a *:30011->3001/tcp
Just last month Docker version 17.12-ce somehow stopped working from one installation to another, causing a break in our install system. Docker probably rebuilt and reused the version number 17.12 when fixing some bug, thinking they did not change a behavior but the did. We do not want this kind of uncertainty in our production system.
Application deployment and configuration changes using Git
Some organizations use software like Jenkins Pipeline to deploy there applications. For us the deployment is a core component. It is the IT departments business logic, the sum of everything we have learnt during the years, and what we know we need to implement to be leaner and faster. So investing in a this was natural. The cost of changing human behaviour, and learning new tools and processes are greater than investing in code. Our goal is to keep things unchanged for the developers. A developer should focus on building value, not learning a deployment process.
When the developers pushes code to Git it is tested, pushed to a Docker registry and installed, just like it always has been at KTH. The same procedure should apply when you wish to deploy a new configuration. Therefore configurations are version controlled and deployed when doing a git push as well. All this put together we decide to build our own deployment demon called Cellus.
Our deployment process Cellus is basically a demon that every 15 seconds does a git pull from a common git repository containing all applications, to see if any changes have been made. Applications are just Docker Compose configurations files. If anything should be deployed, or updated, that application is deployed to the specified Docker Swarm cluster, where the cluster is the directory name where the configuration file is located.
In our deployment demon we have also implemented SemVer so that newly build applications can be deploy, or not deployed in a cluster depending on human readable versions. For example “kthse/places-api:~0.1.0” means that places-api:0.2.3_3f9f245 will not be deployed. This way the product owner can install new major versions of a software in stage, but keep the old versions running i production.
The repository is a file structure looking like /app-name/cluster-name/config-files.
- /application-name |_ /stage | |_ /docker-compose.yml # App configuration for the stage cluster | |_ /secrets.yml # Ansible-vault encrypted secrets (merged with docker-compose.yml on deply) |_ /production |_ /docker-compose.yml # App configuration for the production cluster |_ /secrets.yml # Ansible-vault encrypted secrets (merged with docker-compose.yml on deply)
Apart from deploying applications Cellus also does KTH specific configuration using Docker labels. As a developer you can add information like which Slack channel deployment information should be sent to (mandatory setting, config errors etc) , or specify a specific url used for heartbeat monitoring. More information about how we use Slack in a later post.
# # Sets what slack channels deployment messages should be posted to. # Multiple channels can be provided by using commas: "#channel1,#channel2" # # i.e: - se.kth.slackChannels="#team-kth-webb-build" # - se.kth.slackChannels="#team-pipeline-build,#pipeline-logs" # # Set a specific url to use for key-word monitoring. The url must be provided in # full with protocol and _monitor endpoint. The endpoint will be polled for # "APPLICATION_STATUS: OK" so make sure you're actually pointing to a URL that can provide this response. # Defaults to: https:app.kth.se/app-name/_monitor # # i.e: - se.kth.monitorUrl="https://www.kth.se/path-to-app/_monitor" # - se.kth.monitorUrl="https://api.example.com/api/app-name/_monitor"