Show HN: StableBuild – make any Docker container deterministic
stablebuild.comHi HN!
I've posted this a few weeks back without much HN traction - today we've added a free community tier, so anyone can try it out.
TL;DR: We’ve launched StableBuild, a new tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
As the CTO at an ML startup w/ 75 people (https://edgeimpulse.com/) I’ve grown incredibly frustrated with non-deterministic builds. Last year basically every week one of our containers (we have 40+ unique ones in prod) would stop working properly because some dependency was updated or removed. This ranges from Nvidia deleting cuda base images from Docker Hub, to Chromium being removed from the Ubuntu package registry in favor of the snap version, to pandas 2 being published with breaking APIs - while everyone just depends on e.g. pandas>=1.4.
This has been super disruptive because builds break for no apparent reason: someone pushes some unrelated code change, a container needs to be rebuilt, now it gets the latest dependencies => boom, either a compile error or an integration test fails. Many times this even blocks deployment. If the build system has decided that a container on master needs to be rebuilt, we can’t deploy the complete system if a dependency has shifted. And, fixing this naturally falls on the most senior engineers.
Anyway, to fix this I’ve funded (together w/ my Edge Impulse cofounder) StableBuild. It’s a set of mirrors and registries that let you easily freeze and pin Docker images, apt/apk packages, Python packages, and arbitrary files and URLs from the internet. It currently consists of:
* A custom pull-through cache for Docker Hub, that makes any image pulled immutable. Protects against updated or removed images; and as a nice byproduct also bypasses pull-rate limits in Docker Hub.
* Full daily copies of the Ubuntu, Debian and Alpine package registries + the most popular PPAs; so you can pin to a specific date (give me the package registry as it was on 2023-12-15). Essentially what snapshot.debian.org does, but fast and highly available (and for more repos).
* Full daily copy of the PyPI registry, so you can also pin to a specific date. This has been super useful for resurrecting old Python code. Any Python example w/ dependencies is bitrotted the moment it gets published - StableBuild’s historic registry helps tremendously (see https://docs.stablebuild.com/mirrors-and-caches/pypi-mirror#...)
* A generic file / URL cache for arbitrary things you need to pull from the internet during builds.
This has all been in production with SB’s first customers and has basically eliminated random build failures due to changed dependencies for them. Naturally you still want to upgrade dependencies (security patches are nice!) - but you can do it at their own pace, rather than whenever a container rebuilds.
StableBuild is now available for everyone. There's a free Community tier (since today) that gives free access to all services and mirrors (although with a hard 15GB/month traffic limit), and commercial pricing starting at $199 (cheaper than running a high-available apt mirror on AWS - which we used to do at Edge Impulse).
Would love to hear people's thoughts <3
Sign up: https://dashboard.stablebuild.com
Docs: https://docs.stablebuild.com Nice to see a free tier here! I've been using SB for some time now to prevent our python machine learning packages and containers from breaking. We're super happy and had zero issues so far. Private repositories would be something to wish for though :) Keep up the good work! how is this different from using Earthly and a private Nexus instance? Not sure about Earthly as I’ve never used it, but for Nexus and other artifactory solutions around package repos: 1. Don’t need to set up anything. No need to set up an apt repo and push packages or configure a mirror, because StableBuild already caches the complete registry and thus has everything.
2. Don’t need to think about the complete package list when pushing files to artifactory. Have packages cached from 3 months ago, now want to add another one? Oops, not in Nexus, and the current versions in the Ubuntu package registry are not compatible with your cached versions -> need to update the full dependency tree.
3. Integration is trivial. Three lines to your Dockerfile and done.
4. Can retroactively fix things. Knew that this container built 4 weeks ago? Ok use that as a pin date -> fixed. Then there’s some stuff, like the immutable Docker pullthrough cache and history pypi mirror that I haven’t seen before (but I’d like to learn if others are doing this :-)).