Checksum mismatches on .tar.gz files · community · Discussion #45830

I think the comment you linked says that only your own release archives will be stable, not the source code archives generated by github. They explicitly say the source code archives are not guaranteed to be stable.

That cannot be said, however, for repository code download archives found at archive/v6.0.4.

I guess the part that's unclear to people is what was meant by repository release archives

Both:

• archive/v6.0.4.tar.gz
• archive/refs/tags/v6.0.4.tar.gz

(when fitted into the url template: https://github.com/user/repo/{link_location})

are generated by github. Currently, they both redirect to the same, third location:

• https://codeload.github.com, an internal subdomain of github, followed by the repository slug user/repo, followed by tar.gz/refs/tags/v6.0.4

I am not sure what the original github employee who stated this meant. Perhaps it meant that Github reserves the right to change the internal implementation of archive/v6.0.4 and make it do something else instead.

I think the comment you linked says that only your own release archives will be stable, not the source code archives generated by github. They explicitly say the source code archives are not guaranteed to be stable.

Following up on @eli-schwartz's reply, what the comment stated is that archives accessed via that URL template would be stable. This means source archives that are automatically generated for a release/tag.

I have only observed changes in the compressed archives. If I gunzip a file downloaded today versus one downloaded at some time in the past, the uncompressed tar archives are identical. The compressed (gzipped) archives are different.

This is my experience too. The ungzipped archives are identical.

The problem with this, is that this approach is commonly used to depend on third party repos releases. Not something that you can upload the tarballs yourself to the releases pages of your dependencies, and maintainers have been relying on these tgz as good for their users to consume, I think this was completely unknown, and the result is the ecosystem around github releases is massively broken now.

the problem is that people assume a guarantee that doesn't exist.

neither git nor GitHub guarantee stable hashing, but everyone assumed it was stable.

it is the assumption that is wrong here, not the actual behavior.

However github support had provided some level of guarantee for stable hashing for some urls: bazel-contrib/SIG-rules-authors#11 (comment)
Also github had previously broken this and rolled it back which at least implies some intention for providing stable hashing (since the rollback was in response to community reports of sha256 changes).

TensorFlow is going to solve this, using our own custom mirror.

In general we do, but some deps were not mirrored in a while and there are a few transitive deps not captured.

@eli-schwartz In Nix, a Fixed Output Derivation is a directory or file whos contents are hashed. If the corrupt tarball produces a different directory, it's not going to produce the same sha256. I don't understand what you think a malicious tarball is going to do to circumvent this check. The extraction in Nixpkgs is handled inside a sandbox, so you don't have to worry about tar itself exploiting your host system, as explained in https://github.com/orgs/community/discussions/45830#discussioncomment-4825276

I don't understand what you think a malicious tarball is going to do to circumvent this check.

It could extract files to /etc/shadow or ~/.bashrc or ~/.very_important_tokens, for starters.

So... we're back to "this isn't safe, unless further conditions are fulfilled, by the way which sandbox do you use and what security auditing has it been through".

Which is a far cry from "guys, it's simple, just checksum the extracted contents".

Because that is not simple. It adds more work that has to be done, and the code for doing it is more complicated, and you have to start thinking about sandboxes, then integrating it into your workflows.

It could extract files to /etc/shadow or ~/.bashrc or ~/.very_important_tokens, for starters.

How is it going to do that in a sandbox? Assume trust of the sandbox, otherwise it's a pointless discussion since there is no threat model.

You have to start thinking about sandboxes, then integrating it into your workflows.

With Nix, I can't think of one time I've had to think deeply about the sandbox. It's just there, and it's the only way you can use Nix, unless you turn it off, which takes more effort than working with it.

Why don't we reproduce some of these scenarios? Can you help me exploit an older version of tar? I can make a Nix derivation that actually uses this old version of tar, and can receive some malicious content. We can put it up for people to experiment with then.

add @BillyONeal for vcpkg (I am just making sure vcpkg is included ;) )

You probably also want to add eli-schwartz in the meson WrapDB category.

Craig Scott for CMake FetchContent stuff ?

+1, representing EasyBuild here, it would be great to treat this as an opportunity to improve, in cooperation with experienced package managers...

A change to the tarballs served by GitHub also happened back in Sept 2017, and we indeed just dealt with it then (there was no revert), see:

• checksums for source tarballs downloaded from github.com/.../.../archive can change over time easybuilders/easybuild-easyconfigs#5151 (re-opened today since the problem re-emerged)
• Github Archive URL checksums have changed spack/spack#5411
• fix checksums for packages that download from github.com/x/y/archive easybuilders/easybuild-easyconfigs#5162

The latter PR shows that the impact was relatively limited back in Sept'17, but:

• i) This happened only shortly after we starting keeping track of checksums for everything we download;
• ii) We have way more software supported in EasyBuild now (~2,000 unique "titles" in Sept'17 vs ~5,000 today) - I'm sure the same is true for other packaging tools;
• iii) Way more software is obtained through GitHub today than there was in Sept'17;
• iv) We live in a different world now, where supply-chain security is a hot topic (as it should be), and many more people are paying attention to things like changes in source tarballs (for good reason), as is clear from the size of this discussion.

👋 hey there, sorry for the delay in responding. I would definitely like to talk to communities with a dependency on archive hashes. I've started that discussion here. If anyone would prefer email for privacy (including if you'd like to schedule a face to face video chat), you can use my GitHub handle @github.com.

I was just about to quote Yocto manual on this. While Yocto user's are, probably, an insignificant number to GitHub, if more projects follow this advice, ultimately it will probably be a visible increase in load on GitHub.

And yes, this sanity check saved a lot of people a lot of work. Thanks for that.

FWIW: Even this src-uri-bad QA check isn't always enough, I've just noticed one recipe fetching:
https://opensource.apple.com/tarballs/mDNSResponder/mDNSResponder-878.30.4.tar.gz
which is just redirected to these regenerated archives:

$ wget https://opensource.apple.com/tarballs/mDNSResponder/mDNSResponder-878.30.4.tar.gz 
--2023-01-31 10:19:21--  https://opensource.apple.com/tarballs/mDNSResponder/mDNSResponder-878.30.4.tar.gz
Resolving opensource.apple.com (opensource.apple.com)... 17.253.73.201, 17.253.73.206, 2a01:b740:a26:f100::6, ...
Connecting to opensource.apple.com (opensource.apple.com)|17.253.73.201|:443... connected.
HTTP request sent, awaiting response... 302 Redirect
Location: https://github.com/apple-oss-distributions/mDNSResponder/archive/refs/tags/mDNSResponder-878.30.4.tar.gz [following]
--2023-01-31 10:19:21--  https://github.com/apple-oss-distributions/mDNSResponder/archive/refs/tags/mDNSResponder-878.30.4.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/apple-oss-distributions/mDNSResponder/tar.gz/refs/tags/mDNSResponder-878.30.4 [following]
--2023-01-31 10:19:21--  https://codeload.github.com/apple-oss-distributions/mDNSResponder/tar.gz/refs/tags/mDNSResponder-878.30.4
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2479712 (2.4M) [application/x-gzip]
Saving to: ?mDNSResponder-878.30.4.tar.gz.2?

git fetcher to the rescue :).

I'm not that deep into bitbake, but this sounds like a patch waiting to happen - look for GitHub.com in wget's output and issue the QA.

And yes, pretty much git fetcher to the rescue.

look for GitHub.com in wget's output and issue the QA.

That's significantly more complex than current regex on SRC_URI (without the need to actually fetch something) :).

Kind of expected this. I never did dive into the inner workings.

and most of these changes are usually code changes not tarball refresh on github.

Again, with Nix and NAR, you actually know the difference between a tarball refresh and a content change, whereas without you do not know the difference, and it is an opportunity for people to introduce malicious content into tarballs.

You trust the tarball, I trust the contents.

You trust the tarball, I trust the contents.

I don't trust any tarball :).

Who did audit of the tarball contents before NAR hash was updated?

I don't have enough time in the day to audit 1000 tarballs just because their compression scheme changed, with NAR I don't need to.

Again, with Nix and NAR, you actually know the difference between a tarball refresh and a content change, whereas without you do not know the difference, and it is an opportunity for people to introduce malicious content into tarballs.

And again this only helps in this specific case where a lot of tarballs were changed without expected changes in content.

It doesn't help in all other usual cases where the content as well as tarball were changed during package updates.

It doesn't help in all other usual cases where the content as well as tarball were changed during package updates.

Well thanks to nix-diff I can diff the two and find out quite easily.

@eli-schwartz If Git guaranteed git-archive reproducibility we wouldn't need such additional scripts, right?

I wish there was a way to get the "cool kids" compressions working in a reprocudible way though...

git archive --compression-plugin 😂

I could at least get the git repo

AFAIK, git commit hashes are hashes of the content, so if you know the commit hash and cloned the repo, you don't have to do anything more.

I wish there was a way to get the "cool kids" compressions working in a reprocudible way though...

I believe zstd actually has pretty strong reproducibility gurantees if you use the exact same version.

AFAIK, git commit hashes are hashes of the content, so if you know the commit hash and cloned the repo, you don't have to do anything more.

Yeah but I want to be able to verify an archive hash before I extract it. I don't want to extract to verify its contents. I'm potentially not even interested in the contents, just in why a hash changed. As long as there is a "recipe" to recreate the archive, I'd be totally fine with changing archive hashes.

I think I'd be even better if GitHub published a list of hashes of git archive archives for every commit. Considering the actual commit frequency of even the largest projects it may not even be that computationally expensive. This way one wouldn't even have to download archives 😆

I believe zstd actually has pretty strong reproducibility gurantees if you use the exact same version.

Thanks for this info. Thinking about it, as long as a compression algo is deterministic, it shouldn't even matter whether it's reproducible between versions or not. It only matters whether it is possible to recreate the archive somehow. I think as long as local timestamps etc are not part of an archive, everything goes.

Different needs then.

Re: determinism in compression algos. Everything's fine and dandy when you're single threaded. Multithreading introduces indeterministic factors, especially if you don't know how many threads will be used. And all the "cool kids" compression software is multithreaded nowadays.

Different needs then.

I'm actually surprised how difficult this seems to get right for everyone's needs. As SBOMs become increasingly relevant I'd expect there to be even more diverse requirements. Seems quite hard to get this right for everyone.

Thanks for pointing out the multithreading issue. I wasn't aware, but it totally makes sense that this could produce nondeterminism.

@vtbassmatt I think it's just the repository name, right?

Correct! I misremembered what I had seen Friday night. Good catch.

The point is that the hash of the archive is critical to maintain the same, as downstream projects check the hash and reject the archive if it changes. The directory name is less important as it's stripped by strip-components.

Therefore, it's more important to keep the hash the same rather than keep the directory name consistent. So it would be best if there was some way to be sure that the directory name would be constant even if the repository is renamed, like what happened with Docker/Moby.

Or that can be considered a docker bug, manually reverified, and you can do a one-time hash update. It's semantically no different from cases where a project force-pushes a tag with different contents (and I've dealt with those and had to do full-tree diffs to verify that the code changes were benign).

Reminder: repository renames aren't guaranteed to maintain any links, although it will redirect for as long as the old repository name isn't reused. This is very frequently an issue when a person has a private project, then moves it to an org, and then re-forks the project to their personal account a month later. Tons of redirects that were working and "stable", post-move, break after the fact. And orgs can and do reuse the names of projects that they renamed.

@eli-schwartz We (buildroot) cannot retroactively change hashes in previous releases. In the case of a hash mismatch we would mirror the original file. Ideally users would not have to trust that mirror and could instead depend on the download from GitHub not having a hash mismatch.

Still not this GH issue. Also boost moved tags to another revision, instead of making a new patch release.