Using shallow-git tarballs for CI

· Konstantin Ryabitsev ·

3 min read Original article ↗

If your CI system starts every job with git clone --depth=1 from git.kernel.org, then you're probably going to get banned. Sorry, not sorry. Every shallow clone makes git build a fresh pack, compress it on the fly, and then keep it in RAM until you're done downloading — and it has to do this again for every job in your swarm.

I've previously recommended that you run your own mirror instead, but we now also offer an alternative: every night we publish shallow, single-branch tarballs of mainline and the stable branches here:

https://www.kernel.org/pub/linux/kernel/shallow-tar/

These are tarballs of the shallow clone of the git repo, precisely for CI use. They are also fronted by a CDN, so they should be cached at your local Fastly PoP.

Important safety warning: using a .git directory that somebody else made bypasses a lot of git's defences, so we recommend a few extra safety steps (they are listed in the .git/shallow-tar.readme inside each tarball):

rm -rf .git/hooks .git/config .git/objects/info/alternates
git init
git remote add -t [branch] --no-tags origin [the URL you trust]
git fsck

To make this easier, we have a wrapper script that does all of it for you. Grab it from here:

https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/plain/get-verified-shallow-tar

It should work “out of the box”:

./get-verified-shallow-tar torvalds master

This checks the signed checksums we publish, downloads the newest tarball for that branch, runs the safety cleanups, and then fetches and checks out the latest tip. You end up with ./linux, ready to use for building.

For stable, you need to specify the exact branch that you want:

./get-verified-shallow-tar stable linux-6.12.y <commit>

You should always pass the commit SHA. The tarballs are signed only by the kernel.org autosigner, which just attests that these are the bytes kernel.org published. Kernel.org has been breached before, and you should assume it can happen again, so don't blindly trust content you get from us. The commit ID is what really tells you that you got what you expected. The CI system usually knows the exact commit it wants anyway.

For CI, you probably don't want to look up the signing key on every run. Export it once:

gpg --export autosigner@kernel.org > keyring.gpg

and set USEKEYRING at the top of the script to its full path.

To update the tree at any point, use this command:

git fetch --depth=1 origin <branchname>
git checkout FETCH_HEAD

Do not use git remote update inside a shallow checkout, as it will proceed to download the whole history, which for mainline is about 2.6 GB and potentially even larger for stable. It'll take ages and probably get you banned, because this is very heavy server-side.