From Swamp to Swarm

Solving Infrastructure Scaling at a Fintech Startup

By Jacob Henderson.

The Challenge

What happens when you want to wrap your head around your company’s software architecture, but have an API application with 30 different possible “mission modules”, 4 databases, 3 frontend web apps, and an nginx load-balancer?

Well, if you’re like me, you probably have a stiff drink first.

It’s really not as bad as it sounds. The goal was to simplify and automate Wyre’s deployments in order to control our growing number of platform dependencies and microservices. We succeeded through a combination of long-term planning, extensive testing, and an emphasis on putting in the necessary work and time to stay simple.

This is because keeping infrastructure simple does take work, and in ways that can’t be measured from a pure product perspective. Even worse, there’s sometimes a prevailing bias that complexity is good, that the disorder which emerges as a byproduct of the daily grind is an indicator of progress. Unfortunately this is often illusory, especially since complexity generates its own workload. Nowhere is this more true than inside the infrastructure layer of a tech startup aiming for maximum flexibility.

Stepping Back: The Goals

When we assessed what we wanted, we realized that our “wish list” had the same things as everyone else:

1. No downtime for our web applications, even during scheduled updates.
2. As few dependencies as possible.
3. Daemonization of our core applications (auto-start, restart).
4. Isolation of our apps for the purposes of both security and configuration.
5. Load-balancing, both internal and external.
6. Greater simplicity of deployment, updates, and maintenance.

Each individual item can be accomplished easily with isolated tools. The combination, however, is present only in a few offerings, many of which require heavy customization. We looked at Kubernetes, Azure, and (temporarily used) a common custom Docker solution (Phase II). All three of these solutions would have involved large code changes and heavy custom configuration that would be infeasible at scale.

The remaining “out-of-box” solution was Docker Swarm mode, a new (as of late 2016) integrated platform for managing app containers, routing, load-balancing, health-checking, service dependencies, and even basic configuration. Essentially, all of these components (which we had previously managed separately) could be encapsulated and automated by Docker Swarm.

Smoke Test (Phase II)

Whenever infrastructure gets moved around underneath a series of pre-existing applications, several basic questions need answered:

1. Will critical app function remain unchanged after infrastructure changes?
2. Does the new infrastructure have any software dependencies, and if so, are these dependencies liable to break as they are routinely updated?
3. Do the maintainer(s) of the infrastructure software appear prepared to maintain the software for as long as it will be used?
4. Is the new infrastructure worth the necessary changes to apps, scripts, and processes? It is ultimately less expensive to maintain in terms of time, money, and resources than the old?

For our initial test, we altered our staging environment to utilize a Docker Swarm cluster with custom orchestration. All existing applications were each containerized inside a docker service, with orchestration and tracking of containers occurring using Consul, Registrator, and HAproxy. In order:

1. A Consul cluster is created to keep track of all services and perform health-checks.
2. Registrator runs on each host, finds all the running Docker services, and sends their names and private addresses to Consul.
3. HAproxy, a load balancer and proxy, retrieves the service list from Consul and creates load-balanced routes for each one.

Whenever a service was created and provisioned onto the Docker Swarm using a V2 docker compose spec file, the Consul/Registrator/Haproxy stack would register it and allow the service to scale (ie load-balance via HAproxy) to as many containers across as however many hosts were in the Swarm.

Greater Simplicity

We now understood that we could containerize our app services and get Docker Swarm to run them with the help of some third-party services. However, the result was only marginally less complex to deploy, and required training on three different new services in order to orchestrate correctly. We obviously knew we could do better.

Our lucky break came in January 2017 with the release of Docker Swarm 1.13. The supported features matched our requirements exactly. No external services required to load-balance, manage services, or health-check. Literally one command to deploy an application stack. Rolling updates for containers. And last but not least, the ability to match our prior application behavior exactly, but with greater ease of control and maintenance.

The final production stack looked like (from highest level of abstraction to lowest):

1. Docker-Machine: Manage all hosts at a top level, including ssh, lifecycle, and the settings of the hosts’ docker daemons which make up the Swarm.
2. Docker Swarm: Each host runs a docker daemon that communicates to all other docker daemons in the swarm. One host, the manager, allocates container tasks to all other hosts. An administrator issues commands to the swarm manager to provision Docker Stacks.
3. Docker Stack: A docker stack is a specification of services (and how they communicate) contained in a Version 3 Docker-Compose file. When deployed on a swarm, a stack file produces containers.
4. Docker Containers: A container is a unique environment in which an app can run isolated with its own settings and packages. This is the lowest level of abstraction in the docker ecosystem.

All that was required outside of these components were our databases, an Amazon elastic load-balancer (to map our Swarm ports to domain names), domain names, and an SSL certificate. Services could then be scaled up with the docker service scale. Easy!

Final Thoughts

As for the current state of Wyre, we’re now down to one SQL and one Timeseries database, 2 frontend services, and an API service, all running as micro-services on the Swarm.

It’s been a long 4 month journey from our first basic docker cluster to a simpler, production-ready system. The time investment, however, has been invaluable, considering we’re now ready to scale in a manageable fashion. Over the course of our work we’ve learned a lot, and want to make the lessons available. A full running example of the new Swarm system accompanies this post. Our hope is that instead of being forced to parse obtuse, inconsistent docker documentation or re-learn hard lessons, someone else can pick up what we’ve learned and do it better!