Managing application server dependencies with aptfile

A common pattern in application development is to create a file that contains dependencies for running your service. You might be familiar with some of these files:

• package.json (node)
• requirements.txt (python)
• Gemfile (ruby)
• composer.json (php)
• Godeps.json (golang)

Having a file that contains dependencies for an application is great¹ as it allows anyone to run a codebase and be assured that they are running with the proper library versions at any point in time. Simply clone a repository, run your dependency installer, and you are off to the races.

At SeatGeek, we manage a number of different services in various languages, and one common pain point is figuring out exactly how to build these application dependencies. Perhaps your application requires libxml in order to install Nokogiri in Ruby, or libevent for gevent in Python. It’s a frustrating experience to try and setup an application, only to be given a Cthulu-like² error message about how gcc failed at life:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

Installing collected packages: gevent, greenlet
Running setup.py install for gevent
building 'gevent.core' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -I/opt/local/include -fPIC -I/usr/include/python2.7 -c gevent/core.c -o build/temp.linux-i686-2.7/gevent/core.o
In file included from gevent/core.c:253:0:
gevent/libevent.h:9:19: fatal error: event.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
Complete output from command /var/lib/virtualenv/bin/python -c "import setuptools;__file__='/var/lib/virtualenv/build/gevent/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-4MSIGy-record/install-record.txt --install-headers /var/lib/virtualenv/include/site/python2.7:
running install

running build

running build_py

running build_ext

building 'gevent.core' extension

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -I/opt/local/include -fPIC -I/usr/include/python2.7 -c gevent/core.c -o build/temp.linux-i686-2.7/gevent/core.o

In file included from gevent/core.c:253:0:

gevent/libevent.h:9:19: fatal error: event.h: No such file or directory

compilation terminated.

error: command 'gcc' failed with exit status 1

----------------------------------------
Command /var/lib/virtualenv/bin/python -c "import setuptools;__file__='/var/lib/virtualenv/build/gevent/setup.py';   exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-4MSIGy-record/install-record.txt --install-headers /var/lib/virtualenv/include/site/python2.7 failed with error code 1 in /var/lib/virtualenv/build/gevent
Storing complete log in /home/vagrant/.pip/pip.log.

While projects like vagrant and docker can help alleviate this to an extent, it’s sometimes useful to describe “server” dependencies in a standalone file. Homebrew users can use the excellent homebrew-bundle - formerly brewdler - project to manage homebrew packages in a Brewfile like so:

1
2
3
4
5
6
7
8
9
10
11
12

#!/usr/bin/env ruby

tap 'caskroom/cask'
cask 'java' unless system '/usr/libexec/java_home --failfast'
brew 'elasticsearch'
brew 'curl'
brew 'ghostscript'
brew 'libevent'
brew 'mysql'
brew 'redis'
brew 'forego'
brew 'varnish'

Simply have a Brewfile with the above contents, run brew bundle in that directory on your command-line, and you’ll have all of your application’s external dependencies!

Unfortunately, this doesn’t quite solve the issue for non-homebrew users. Particularly, you’ll have issues with the above approach if you are attempting to run your application against multiple non-OS X environments. In our case, we may run applications inside both a Docker container and a Vagrant virtual machine, run automated testing on Travis CI, and then deploy the application to Amazon EC2. Re-specifying the server requirements multiple times - and keeping them in sync - can be a frustrating process for everyone involved.

At SeatGeek, we’ve recently hit upon maintaining an aptfile for each project. An aptfile is a simple bash script that contains a list of packages, ppas, and initial server configurations desired for a given application. We can then use this to bootstrap an application in almost every server environment, and easily diff it so the operations team can figure out whether a particular package is necessary for the running of a service³.

You can install the aptfile project like so:

1
2
3

# curl all the things!
curl -o /usr/local/bin/aptfile https://raw.githubusercontent.com/seatgeek/bash-aptfile/master/bin/aptfile
chmod +x /usr/local/bin/aptfile

We also provide a debian package method in the project readme for those who hate curling binaries.

To ease usage across our development teams, a dsl with a syntax similar to bundler was created. The aptfile project has a few primitives built-in that hide the particulars of apt-related tooling:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#!/usr/bin/env aptfile
# ^ note the above shebang

# trigger an apt-get update
update

# install a packages
package "build-essential"

# install a ppa
ppa "fkrull/deadsnakes-python2.7"

# install a few more packages from that ppa
package "python2.7"
package "python-pip"
package "python-dev"

# setup some debian configuration
debconf_selection "mysql mysql-server/root_password password root"
debconf_selection "mysql mysql-server/root_password_again password root"

# install another package
package "mysql-server"

# we also have logging helpers
log_info "🚀  ALL GOOD TO GO"

One potential gripe behind a tool like this is that it would lock you into the dsl without being very expressive. Fortunately, an aptfile can contain arbitrary bash as well:

1
2
3
4
5
6
7
8
9
10
11

#!/usr/bin/env aptfile

update

# pull down a debian package from somewhere
curl -o /tmp/non-repo-package.deb https://s3.amazonaws.com/herp/derp/non-repo-package.deb

# install it manually
dpkg -i /tmp/non-repo-package.deb

# continue on

Another issue we found is that tooling like this can either be overly verbose ⁴ or not verbose enough ⁵. The aptfile project will respect a TRACE environment variable to turn on bash tracing. Also, if there is an error in any of the built-in commands, the log of the entire aptfile run will be output for your convenience.

For the complete documentation, head over to the Github repo. We hope you’ll be able to use bash-aptfile in creating a better, faster, and smoother developer experience.