7x faster Docker builds with lazy layer pulling and cache-mount export. — Clipper Blog

Common wisdom for Docker builds to use --mount=type=cache doesn't actually hold when using ephemeral builders (such as GitHub Actions). Because BuildKit doesn't expose any native way to export these caches (not to be confused with the build cache), they get thrown away every build. Additionally, many large images have files in the base that are only used at runtime and not during the build. This is wasteful. Why pull bytes your build doesn't use? With a custom BuildKit driver+registry, I've solved both these problems, with a speedup of >7x when building llamacpp.

Background

First off, a quick explanation for what makes Clipper's underlying format different than Docker.

I'm not going to go into too much on how Docker works, but generally: During a docker build, each command in a Dockerfile (such as RUN apt install vim or COPY --from=build /install /usr/local/) creates a filesystem. That filesystem consists of the files that were added/changed/removed during the build. Once the build is complete, each of those filesystems is transfomred into a layer.

For Docker/OCI, that layer is a a compressed tarball. A simple layer that adds three shared objects might look like this, as one monolithic tar file:

Clipper instead produces a table of contents JSON file holding the metadata as well as separate blobs containing the file data.

(a lot of fields are omitted here as well as some implementation details around small files)

When clipper pull is run with such a layer, the client does one of two things:

1. for containerd (or Kubernetes/Docker on top of containerd), we can directly write a new filesystem into containerd's content store, with no round tripping through other formats
2. for Docker (no containerd) and Podman, we convert back to a tar layer and let those systems handle conversion back to a filesystem

Some differences to note:

• With regular OCI layers, changing the metadata for a file results in a completely new tarball layer with no data shared with the old one. Clipper doesn't suffer from this - a new TOC will be created, which will usually be under a MB in size, and the file data will be shared between the old and new layers.
• All OCI objects are referenced by compressed digest. Recompressing, uncompressing, or just running a build on another machine with a newer compression library will result in a new hash for an object. All Clipper references are by uncompressed digest, even if compression is later applied to an object.

It's somewhat possible to squint and see how this will result in faster pushes and pulls due to better sharing between layers, so let's talk about builds.

The test setup

Scenarios

llamacpp

• Building llama.cpp, with CUDA enabled, and copying the build results on top of a runtime CUDA base layer.
• We bust the cache every build by injecting the current timestamp into the main cpp file.

ARG BASE_IMAGE
ARG RUNTIME_BASE
 
FROM ${BASE_IMAGE} AS build
 
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    rm -f /etc/apt/apt.conf.d/docker-clean && \
    apt-get update && \
    apt-get install -y --no-install-recommends \
        build-essential cmake git ca-certificates curl xz-utils
 
ARG CCACHE_VERSION=4.13.6
ARG TARGETARCH
RUN --mount=type=tmpfs,target=/tmp
    case "$TARGETARCH" in \
        amd64) cc_arch=x86_64 ;; \
        arm64) cc_arch=aarch64 ;; \
        *) echo "unsupported TARGETARCH=$TARGETARCH for ccache install" >&2; exit 1 ;; \
    esac && \
    pkg="ccache-${CCACHE_VERSION}-linux-${cc_arch}-musl-static" && \
    curl -fsSL "https://github.com/ccache/ccache/releases/download/v${CCACHE_VERSION}/${pkg}.tar.xz" -o /tmp/ccache.tar.xz && \
    tar -xJf /tmp/ccache.tar.xz -C /usr/local/bin --strip-components=1 --no-same-owner "${pkg}/ccache" && \
    ccache --version
 
ADD https://github.com/ggml-org/llama.cpp.git#0827b2c1da299805288abbd556d869318f2b121e /src
WORKDIR /src
 
ARG CACHE_BUST
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs/
RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1
RUN --mount=type=cache,target=/root/.cache/ccache \
    export CCACHE_LOGFILE=/tmp/ccache.log CCACHE_STATSLOG=/tmp/ccache.statslog && \
    echo "// bench-mutation ${CACHE_BUST}" >> src/llama.cpp && \
    cmake -B build \
        -DGGML_CUDA=ON \
        -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \
        -DCMAKE_C_COMPILER_LAUNCHER=ccache \
        -DCMAKE_CXX_COMPILER_LAUNCHER=ccache && \
    cmake --build build -j"$(nproc)" --target llama-cli
 
RUN mkdir -p /opt/llama/bin /opt/llama/lib && \
    cp build/bin/llama-cli /opt/llama/bin/ && \
    cp build/bin/*.so* /opt/llama/lib/
 
FROM ${RUNTIME_BASE}
COPY --from=build /opt/llama /usr/local

uv

• Running uv sync on a pyproject with a number of ML packages

ARG BASE_IMAGE
FROM ${BASE_IMAGE}
 
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
 
WORKDIR /app
COPY pyproject.toml uv.lock ./
 
ARG CACHE_BUST
 
RUN echo "$CACHE_BUST" >/dev/null && \
    uv sync --frozen --no-progress && \
    rm -rf /root/.cache/uv
ENV PATH=/app/.venv/bin:$PATH

[project]
name = "ml-bench"
version = "0"
requires-python = "==3.14.*"
dependencies = [
    "tensorrt",
    "accelerate",
    "datasets",
    "scipy",
    "scikit-learn",
    "pandas",
]
 
[tool.uv]
package = false

There are ways to improve both Dockerfiles, but they represent pretty normal usage.

Builders

GHA:

• Default GitHub Actions 4 core Ubuntu builders
• There's some amount of variance between runs in GitHub Actions, depending on what other builders in the same physical hardware are doing (and the exact CPU model you get).
• All GHA timings were pulled from here: https://github.com/clipper-registry/blog-buildkit-benchmark/actions/runs/28127099905

Orin:

• An NVIDIA Jetson Orin devkit sitting in my living room with a Samsung SSD 990 PRO 1TB SSD for storage.
• Wifi speed is around 100Mbps
• All timings generated locally, you'll just have to trust me

Registry + Driver

Docker Hub

• Pushing to https://docker.io
• Using moby/buildkit:latest (v0.31.0 or v0.31.1)

Clipper

• pushing to https://clipper.dev
• Using pre-converted CUDA base layers
• Using clipperregistry/buildkit:v0.31.0-clipper

I measure "pull time" from the start of the build until the last pull is done. I measure export time from the start of the image+cache export until the end of the build. This is just an approximation, but a good enough approximation.

1. Baseline - Clipper vs upstream

Here's the baseline timings:

Clipper is around 16s faster. This is mainly driven by faster pull times - overall, I've found Clipper to be 10-20% faster on fresh pulls when running with fast internet on very fast SSDs.

Clipper was 1.2-3x faster than upstream. Specifically for the uv test, the pull stage was a full minute faster, and export was a minute and a half instead of 20 minutes.

llamacpp

Pull

Why is Clipper faster on llamacpp?

1. It's a little more parallelizable. Docker heavily bottlenecks when hitting a large layer. It's forced to serially fetch and then extract. Clipper's layers are broken up by file, meaning when one large file is extracting, another can be downloading.
2. I always use zstd for compression, and specifically import https://github.com/datadog/zstd, which I've found to have just a bit of a speed boost (10%?) over pure Go alternatives. I don't mind having to compile with CGO, I've wrangled cpp build chains for over a decade, now.
3. I pick higher compression settings by default (9 instead of 3). Clipper compresses on the backend, on ingest, with support for recompression. Future releases will expose controls for picking even higher compression levels, as long as you pay for the compute.

Why was export slower for Clipper?

• Clipper does a bit more work when exporting. It's doing more queries to the server for chunk existance. I've done quite a bit of work to optimize this, but there's still more work to be done.

uv

There's clear wins for export times. Why?

• BuildKit is forced to tar up and export every file. Clipper's driver isn't. Clipper can check to see if the registry already has a large file, and skip copying to a tar, compression, a round trip to disk, and upload. It's especially visible for the local build, ~10 minutes instead of 30 is huge.
• There's some performance wins possible here for Clipper, still, but they require a change to the Clipper layer format. Clipper still uses one large tarball for small files. More on this later.

2. Cache mounts

Overall, that's great, but we can do better, Let's start with focusing on the llamacpp build.

The longest part of the llamacpp build by far is the actual build. We spend 332.5s running what is essentially cmake && cmake build. We even use ccache to reuse results from previous builds! There's a problem, though:

Every time we compile, we cache build results via ccache and then...throw it away.

We attempt to save it (via --mount=type=cache,target=/root/.cache/ccache) but the nature of GHA means we throw away the mount contents every build.

How can we fix it?

1. buildkit-cache-dance
   • Pros
     • It mostly just works
     • It's very configurable
   • Cons
     • It's mainly meant to run in GHA. You can run it in other places, but it needs node (you could probably extract the underlying logic into a shell script, though)
     • GitHub caches are kind of wonky - there's no way to share a cache back to main from a feature branch, or between unrelated feature branches
     • GitHub cache size can fill up fast. 10GB isn't that much. (But at least finally they allow paying money to get more cache.)
     • It won't deduplicate between caches. This has impacts on cost and speed.
     • Depending on the type of thing you're caching, your cache may grow endlessly. Even if your cache size is capped, the nature of caches means you are going to pull data you do not use.
     • It's a little bit slow to export for large caches.
2. Use sccache, ccache's remote_storage, or similar options for other languages
   • Pros
     • Native support for your language
   • Cons
     • Needs infra setup, permissions, etc.
     • Different setup for every language, not all languages and cache types may be supported.
3. Don't use cache mounts at all, instead use SSHFS, a manual cache import/export inside the Dockerfile, a host mount, or something else to mount a network cache during the build, with a build time secret passed in for auth.
   • Pros:
     • It does technically work.
   • Cons:
     • Dockerfile complexity. You must manually mount at every step that could use the mount.
     • Not really usable for some cache types due to lack of locking behavior.
     • Infrastructure complexity. The data has to live somewhere, you need to secure it.
4. Use Depot or another GHA build machine provider
   • Pros:
     • It's supposedly fast.
   • Cons:
     • It costs money, you can't use the free CI minutes GitHub gives you.
   • (At some point I will profile this. I'm sure it's great. I'd be interested to compare GitHub+Clipper vs Depot vs Depot+Clipper.)
5. Host your own infrastructure and use persistent builders that don't throw away cache mounts.
   • Pros:
     • It's fast
   • Cons:
     • Look, it's great if you have an infra team and budget to do this. Not everybody is there.
     • You might still want to use ephemeral builders, so that you don't have to worry about disk space filling up, OS updates, minimal build queuing, etc.

In an ideal world, we would fix this with something that:

1. Doesn't require Dockerfile changes
2. Doesn't require maintaining any infra
3. Scales well to large cache sizes

The simplest thing for Clipper to do would be to just add an argument to import/export cache mounts. This would fit the first two points - it works with existing Dockefiles and you're already using a registry. The last point is a bit trickier, but doable.

What if instead of downloading the entire cache, we only downloaded the files we need? We can expose the cache directory as a FUSE filesystem and download the files as they are read. This is a similar trick used by eStargz and Seekable OCI to enable fast container startup in the cloud.

I'm not going to go into how FUSE works in depth, but at a high level:

• We create a library that exposes a set of callbacks around filesystem (meta)data to the kernel
• The kernel calls into our library any time some other process tries to do an operation like open() read() or readdir()

This allows exposing arbitrary data as a filesystem. You can do a lot of cool things with FUSE.

There are some downsides:

1. Reads will be a bit slower
2. It's additional complexity
3. We're moving some time from pull time to build time. There will be some latency between the requesting the read of a file and actually reading.

I've worked around the last downside a bit by:

1. Prefetch all small files (required by the current Clipper TOC format anyhow, to know the fs structure)
2. Start fetching any medium size files on open()
3. Not requiring the whole file to be downloaded before streaming bytes to read() calls

I've come up with a pretty clever overlay that's essentially this:

At the very bottom, files and other data we already have are put into a real filesystem. In the middle, we synthesize a FUSE layer to pull large files on demand. Finally, on top we have a writable layer swo that we can easily generate a diff.

So...did this actually help? Yes!

Clipper's cache mount export comes out a bit faster than using buildkit-cache-dance, but there's also a hidden advantage here that isn't visible with these numbers: we are exporting the minimum cache size to build llamacpp. A real-world workflow will end up growing the cache up to the maximum. The cache exported here is 140MB, so we can estimate that a full 1GB cache would take about 7x as long, adding an additional ~138s to the build.

There are similar gains for local builds, but they are somewhat masked by the slow download time for the base layer.

3. FUSE all the things

This is great, but...could we go even further? (Hint: Yes)

We now have a bunch of infrastructure in place to handle Clipper layers with FUSE. What if we applied it to not just cache layers but all Clipper layers. The work to do this took longer than I expected (three weeks) but here's the results:

This is a huge win, over 7 times faster than the baseline. The pull time is essientially zero, and the build itself takes only a little bit longer.

Local builds are a bit faster, too: 3x->4x, over two minutes faster.

Getting this right wasn't easy, involving making sure we track where each chunk came from so that we can do a cross-repo mount (essentially a serverside copy from one namespace to another) and ensure we don't accidentally materialize full layers. At serveral points I thought I had a fully working implementation, then discovered that I was silently pushing extra GBs of data.

Futher work

• I could have stronger statistical evidence for the speedups. I'm not cherry-picking them, but they are essentially just the "latest" number from the last time I ran before writing this post.

• I should also run this on more repositories. Depot lists half a dozen different repositires that they speed up, I could profile on other repos, and maybe gain some users. There are definately repos that I would speed up more than 7x. If you have a repo that you'd like to use Clipper to accelerate, please reach out.

• The FUSE setup could be better tuned. A file being open()ed doesn't neccessarily mean that it's about to be read from. I also need to expose controls for parallelism.

• While Clipper does support slicing a file into chunks, I'm not currently taking advantage of it in BuildKit. There's some tension between more chunks leading to better sharing between files (and being able to fetch less data for FUSE) and more chunks causing more HTTP requests, leading to more server load and higher latency. The answer is probably to add an endpoint that can do bulk operations to generate redirect URLs for chunks.

• There are several changes I want to make to small file handling that make cache export still slower than it needs to be. The uv export could be made near instant if I was able to split apart small file bundling by package, and pull metadata for the files up into the toc.

• I'd previously resisted making a remote snapshotter for Clipper, due to it being useless for the original use case, robotics (robots don't want to start running on an update that's half downloaded!) However, there are some genuine uses for lazy pulled images in other spaces, so I'll likely extract it from BuildKit out into a standalone installable daemon. I also plan on looking into EROFS as other snapshotters use it to improve read performance.

• If a new chunk is pushed in both image and cache, I don't currently dedupe between them.

• While exporting regular OCI images from the Clipper Driver is definitely possible, it won't preserve base images hashes. Fixing this requires tracking the providence of converted images. The CLI does this for conversions, but this isn't persisted out to the registry.

• All of this work makes me wonder if I could make a generalized cache import/export for directories for non-Docker builds. There are loads of projects out there that do similar tarball dances to export to GitHub Actions caches that I could speed up.

Final words

This took two months of work, alongside which I was also working on other things (reflink support on CoW filesystems, performance and QoL upgrades, fundraising, getting over a massive cold, etc). It's a huge relief for it to be out. I'm extremely pleased with the results.

Please give it a try. Clipper is looking for users!

Reach out to [email protected] or join our Discord if you have questions!