gödel: A Build System for the Go Programming Language

Press enter or click to view image in full size

gödel is a build tool for the Go programming language. It is the primary development and build tool for over 100 Go projects at Palantir and is available under the Apache 2.0 license on GitHub.

Get Palantir’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

gödel’s architecture is similar to Gradle’s and comprises a bootstrap executable (godelw) and a configuration directory (godel). Once gödel has been added to a project, the typical development cycle is:

./godelw idea gogland to generate a Gogland project
./godelw verify to run checks and tests
./godelw publish to build and publish binaries

gödel is written in Go and is of course built by itself. It delegates to standard Go commands and tools to execute its tasks. gödel simplifies the build system setup, ensures a consistent and predictable development experience, and allows organizations and communities to standardize their build workflows by separating build logic from build configuration.

Motivation

When we started to use Go for some of our projects at Palantir, we struggled to provide a consistent experience for developing, testing, building, and publishing. Developers would manually run a set of static checks and tests on the code, then commit code to a version control system running continuous integration. The CI system would then run the same set of static checks and tests, build executables for a variety of target platforms, create distribution archives, and upload them for distribution. Maintaining consistency between the different development environments and the CI system proved to be a challenge, especially as teams started to grow. We started to adopt tools that performed static checks (such as errcheck), but because they were installed using go get (which installs the newest version at the time it is run), the results could differ depending on when a developer set up their environment.

As the complexity of our projects increased, we introduced scripts to encode the verification and publishing workflows—and then promptly ran into the classic problems posed by scripts: the scripting logic became more and more convoluted and per-project configuration became intertwined with the build logic itself. When teams started new projects, they would copy scripts from other projects and make slight modifications. This process was prone to errors and made it hard to push out updates globally at a later time.

These issues had an impact on our development velocity, so we decided to tackle what we viewed as the fundamentals of the problem. We wanted a build system that had the following properties:

Stable: Each project declares the exact set of checks and operations that are run at a project level, versioned together with the project source code
Modular: The project maintains the build system configuration, not build logic
Maintainable: The build logic is maintained by a central team and can easily be updated across projects
Consistent: Development environments and build behavior are consistent across projects

We took a look at various open-source Go repositories, but did not see a satisfactory solution. Many of the smaller projects had CI scripts that would go get the checks and depended on go get as their distribution mechanism. Larger projects often used makefiles and scripts. Gradle is a great build system for the JVM ecosystem, but it looked like we would have to build out multiple plugins for Go support, and requiring a JVM to build our Go projects was not appealing either.

gödel as a tool for declarative builds

Using the above list as a guide, we started on gödel during Palantir’s bi-annual Hack Week in 2016. Since then, it has matured to become the standard build system for all Go projects at Palantir. Its design borrows from Gradle: the gödel binary encapsulates the logic for performing a set of build tasks (such as syntax checking, compilation, test execution, packaging, publishing, etc.), and gödel-enabled projects specify the desired gödel version and project-specific configuration for the build. gödel guarantees that the execution of ./godel verify on a given Git commit of a project performs the exact same set of verification tasks regardless of where the command is executed, be it a developer laptop or a CI system.

gödel can be enabled for a project by adding the godelw script and godel configuration directory to a project. Invoking the ./godelw script causes the proper version of the godel binary to be downloaded and installed locally. Once gödel has been added to a project, the typical development cycle is as follows:

Run ./godelw idea gogland to generate a Gogland project (configured to run ./godelw format on save)
Write code
Run ./godelw verify to verify that the project passes all of its checks (static code analysis, tests) and enforces proper state for things like license headers and go generate tasks
Configure the CI system to run ./godelw publish to build and publish the distribution for the project

The gödel tutorial provides a full walk-through of gödel, including setting up a project and making use of all of gödel’s features. From the projects we have done so far, we have observed the following main benefits:

Increased velocity for new projects. Developers can set up new projects that build and publish artifacts more quickly. In the past, they had to manually copy and edit scripts from other projects, which was an error-prone and lengthy process.
Faster and greater adoption of build logic updates. It is now possible to determine the build logic used by a project by examining the version in a project’s godel/godel.properties file. Better yet, we can automatically push build system upgrades to all projects. Before gödel, pushing out updates to build logic across projects was a painful and time-consuming process, and consequently, many scripts fell out of date.
More consistency when re-running builds in CI over time. CI builds were not stable over time because our build scripts used go get to install the check logic. Using gödel to lock in the version of the checks and logic being run has increased the reproducibility of our builds over time.
More developer collaboration between projects. Because our projects have standardized on gödel, it has become much easier for developers to quickly check out and contribute to different projects, increasing the amount of cross-product collaboration and drop-in contributions and pull-requests.
Compounding of knowledge and value. With gödel, build system improvements can be easily shared with our Go community. For example, when we wanted to create a task that would allow projects to build Docker images of their products, it was easy for a developer to add this feature to gödel because it already had a notion of products and their distributions. The manner in which all of the build-specific configuration is centralized in the godel/config directory has also made it easy for teams to examine each other's configuration to determine how to do certain things. In the past, such configuration was often intertwined in script logic in a way that made it hard to read and apply to other projects.

Conclusion

Using gödel has vastly improved the Go development experience at Palantir. It has simplified bootstrapping of new projects and significantly reduced the complexity of managing large projects. gödel provides a consistent experience between developers and CI systems. It also strikes a balance between having a low floor and a high ceiling: the default configuration of gödel is sufficient for building, distributing, and publishing a standard project, but gödel also has enough flexibility to support building large, multi-product projects that create multiple distributions and have extensive customizations.

We believe that everyone can benefit from the general principles that are enabled by gödel, and hope that others in the Go community may find it useful. We are confident that gödel can provide significant value in its current form, and have released it as version 1.0.0. However, the current architecture of gödel does not allow it to be as flexible as it could be: since all tasks must be compiled into gödel itself, it is not possible to dynamically customize the linting checks that are available to it, or to add custom tasks to gödel without modifying the core repository. The next major step in gödel’s development is to make the architecture more pluggable so that tasks can be added or customized dynamically.

We look forward to feedback and welcome contributions!