Working draft
GitHubhaampie/tar-minefield DOI10.5281/zenodo.23617964
This project provides an extensive test suite and documentation for tar file extraction, with the goal of ensuring that implementations extract tar archives the same way.
There are many implementations of tar, from command line tools to libraries for programming languages. Because the tar file format leaves room for interpretation, different implementations can extract certain crafted tar files in different ways.
This is a security concern for file exchange: for example, when a security scanner and the extractor use different tar implementations, the scanner could inspect other files than the ones that end up on disk.
By assigning identifiers to these ambiguities, and providing test cases and usage data, this project aims for tar implementations to converge on strict parsing and identical behavior.
This project proposes a default policy that rejects the vast majority of ambiguities in tar files that could make implementations extract different files, while ensuring that nearly every source tarball already published can still be extracted. Of the 82,586 source tarballs of Spack, nixpkgs, Guix and live-bootstrap, and samples of PyPI, crates.io and RubyGems, it accepts 99.80%.
Comparison of different tar implementations¶
This table shows how different tar implementations behave on each test case. A green ✓ is the
strict behavior that the default policy asks for; click a row for details
about the case.
Each case has a permanent identifier, such as F15, whose letter is its category:
- F framing: which members an archive has.
- N names: what path each member has.
- K kinds: the kind of each member: file, directory or link.
- S order and state: how members combine into a directory tree.
A case's page shows its archive block by block, what every implementation makes of it, and how common it is in release tarballs. An implementation's page, linked from its name above the matrix, lists the cases where it does not do as the policy says.
Policy¶
This section defines a working draft of the default policy1, which accepts, tolerates or rejects each case below, based on data from the 82,586 distinct release tarballs of Spack, nixpkgs, Guix and live-bootstrap, and samples of PyPI, crates.io and RubyGems.
The prevalence column is the share of these tarballs that have a case: a first-order approximation of how common it is in general.
1 A policy can be stricter and also reject the tolerated cases, at the cost of rejecting more published tarballs: some tolerated cases are common, such as numbers padded with spaces (F01), and others rare. That suits special-purpose tools, such as an extractor for a single ecosystem, where it can be checked against every tarball ever published there.
To accept¶
| identifier | name | what it is | prevalence |
|---|---|---|---|
F17 | pax-global-comment | The global pax header with only a comment, as git archive writes it. | not measured |
F28 | pax-newline-in-comment | A newline in a pax comment, which split on lines leaves a size record. | none |
F31 | pax-xattr-not-utf8 | An xattr in a pax header whose value is not UTF-8, as libarchive and star write them. | 0.04% |
F57 | pax-newline-in-comment-path | A newline in a pax comment, which split on lines leaves a path record. | none |
F58 | pax-newline-in-comment-then-size | A newline in a pax comment, which split on lines leaves a size record before the last one. | none |
F59 | pax-nul-in-xattr | A NUL in the value of a pax xattr, before a path record. | 0.04% |
K07 | dir-name-empty-with-prefix | A directory with an empty name field and a prefix. | 7.65% |
N01 | gnu-long-name | A name over 100 bytes, in a GNU long name. | not measured |
N02 | name-fills-field | A name of exactly 100 bytes, which fills the name field without a NUL. | 9.53% |
N03 | pax-long-name | A name over 100 bytes, in a pax path. | not measured |
N04 | ustar-prefix | A long name split between the ustar prefix and name fields. | 7.65% |
N27 | pax-long-symlink | A symlink with a long name and a long target, in a pax path and linkpath respectively. | 0.04% |
N28 | pax-long-hardlink | A hardlink with a long name and a long target, in a pax path and linkpath respectively. | 0.04% |
N42 | gnu-long-name-dir | A directory with a long name in a GNU long name, then a file. | not measured |
N43 | pax-long-dir | A directory with a long name in a pax path, then a file. | not measured |
N44 | pax-path-name-empty | A regular file with an empty name field and a pax path. | not measured |
N45 | gnu-long-name-field-slash | A file with a GNU long name and typeflag NUL, whose name field ends in a slash. | 4.38% |
N46 | pax-linkpath-linkname-empty | A symlink with an empty linkname field and a pax linkpath. | not measured |
N47 | gnu-long-link | A symlink with a long target, in a GNU long link name. | 0.04% |
S01 | tree-dir-after-contents | A directory after its contents. | 1.22% |
S02 | tree-duplicate-dir | The same directory twice, as tar cf x.tar d d/sub writes. | 0.19% |
Tolerated¶
| identifier | name | what it is | prevalence |
|---|---|---|---|
F01 | number-leading-space | Numeric fields padded with spaces rather than zeros, as old tars wrote them. | 0.99% |
F02 | end-data-after | Data after the end-of-archive marker that is not a header, which is ignored. | 0.03% |
F04 | end-missing | No end-of-archive marker: the archive ends after a complete member. | 0.03% |
F05 | magic-version-nuls | The POSIX magic with NULs for its version, "ustar\0\0\0", as some Java archivers write. | 0.07% |
F14 | pax-size-disagrees | A pax size that disagrees with the header's, so that a reader of the header's misses a member. | none |
F18 | pax-global-comment-after-members | A global pax header with only a comment, after the first member. | 0.10% |
F32 | checksum-leading-space | A checksum padded with spaces rather than zeros. | 0.99% |
F42 | pax-comment-not-utf8 | A pax comment that is not UTF-8. | none |
F43 | pax-global-comment-not-utf8 | A global pax comment that is not UTF-8. | none |
F55 | type-contiguous | A contiguous file, typeflag 7. | none |
F56 | number-size-base256 | A size of 512 in base-256. | none |
N05 | name-dot | A member named ./, the root of the tree, as tar cf x.tar . writes first. | 0.08% |
N06 | name-leading-dot-slash | The name ./a, which is the path a. | 0.30% |
N07 | symlink-absolute | A symlink to an absolute path, created as it is. | 0.11% |
N15 | name-control-character | A control character in a name, here a newline, created as it is. | 0.01% |
N35 | symlink-escapes | A relative symlink whose target leads out of the destination. | <0.01% |
S16 | hardlink-target-spelled-differently | A hardlink to ./a, where the member is named a. | none |
To reject: framing¶
| identifier | name | what it is | prevalence |
|---|---|---|---|
F03 | end-lone-zero-block | An end-of-archive marker of one zero block, not two. | <0.01% |
F06 | checksum-signed | A checksum that only matches as the signed sum of the header. | none |
F07 | end-archive-after | Another archive after the end-of-archive marker. | <0.01% |
F08 | end-truncated-in-member | An archive that ends inside the data of a member. | not measured |
F09 | end-zero-block-then-member | A zero block, then another member: the end of the archive, or not? | <0.01% |
F10 | extended-pax-size-before-gnu-long | A pax size followed by a GNU long name: which of them does the size apply to? | none |
F11 | magic-odd-version | The POSIX magic with a version that is neither "00" nor NULs. | none |
F12 | number-size-gap | A size with a space among its digits, which reads as two sizes. | none |
F13 | pax-newline-in-path | A newline in a pax path, which split on lines is another record. | none |
F15 | pax-size-on-dir | A pax size on a directory, which takes the next header for its data. | 0.01% |
F16 | type-unknown | An unknown typeflag with a size: data, or no data? | none |
F19 | pax-global-between-extended-and-member | A global pax header between an extended header and its member. | none |
F20 | pax-global-size | A global pax header with a size, which frames every member after it, or none. | none |
F21 | pax-size-empty | An empty pax size, which in POSIX removes the size rather than giving one. | none |
F22 | pax-size-negative | A negative pax size, which a member cannot have. | none |
F23 | pax-size-plus | A pax size with a plus sign, +512, which conversions of strings to numbers accept. | none |
F24 | pax-size-leading-space | A pax size with a leading space, 512, which conversions of strings to numbers skip. | none |
F25 | pax-size-not-ascii-digits | A pax size in Arabic-Indic digits, ٥١٢, which some conversions of strings read as 512. | none |
F26 | pax-size-leading-zero | A pax size with a leading zero, 0777: 777 in decimal, as POSIX says, or 511 in octal. | none |
F27 | pax-size-too-large | A pax size of 2^64 + 512, which wraps around to 512 in 64 bits. | none |
F29 | number-size-base256-negative | A negative base-256 size in the header, -512, which a member cannot have. | none |
F30 | number-size-base256-too-large | A base-256 size of 2^64 + 512 in the header, which wraps around to 512 in 64 bits. | none |
F33 | size-on-dir | A size in the header of a directory, which takes the next header for its data. | none |
F34 | size-on-hardlink | A size in the header of a hardlink, in a pax archive, which takes the next header for its data. | none |
F35 | pax-size-on-hardlink | A pax size on a hardlink, which takes the next header for its data. | none |
F36 | pax-record-length-short | A pax record whose length is one byte short, so that it ends just before its newline. | none |
F37 | number-size-not-octal | A size with the digit 9, which is not octal. | none |
F38 | size-on-hardlink-without-pax | A size in the header of a hardlink, in an archive without pax headers: ustar or pax? | none |
F39 | format-pax-after-v7 | A v7 header first, then a member with a pax header: is the x header a header, or a file? | none |
F40 | number-size-no-terminator | A size of 12 octal digits, without a terminator. | none |
F41 | number-size-leading-nul | A size with a NUL before its digits, which takes the next header for its data. | none |
F44 | pax-global-size-empty | An empty size in a global pax header: does it delete the size of every later member, or set it? | none |
F45 | type-gnu-sparse | An old GNU sparse file, typeflag S: the data as stored, or the file it expands to? | none |
F46 | pax-size-on-symlink | A symlink with a pax size: data to skip, or none? | none |
F47 | checksum-wrong-after-first-member | A header with a wrong checksum after the first member: the end, a block to skip, or a header? | not measured |
F48 | pax-record-length-plus | A pax record whose length has a plus sign, +17. | none |
F49 | pax-record-empty-line | An empty line between two pax records. | none |
F50 | pax-size-underscore | A pax size with an underscore, 5_12, which some conversions of strings to numbers accept. | none |
F51 | number-size-plus | A size with a plus sign, +0000001000. | none |
F52 | type-gnu-volume-label | A GNU volume label, typeflag V: a file, or skipped? | none |
F53 | type-gnu-old-long-name | A header of typeflag N, which some readers take for a long name: a name, or a file? | none |
F54 | format-pax-header-without-magic | A pax extended header without magic: a header, or a file? | none |
N48 | magic-gnu-version-00 | A prefix in a header with the magic "ustar " and the version "00": part of the name, or not? | none |
To reject: names¶
| identifier | name | what it is | prevalence |
|---|---|---|---|
N08 | extended-pax-path-and-gnu-long-name | A pax path and a GNU long name for one member, and a third name in its header. | none |
N09 | extended-two-pax-headers | Two pax extended headers for one member, with different paths. | none |
N10 | format-star | A ustar prefix whose end reads as star's times, with star's tar\0 at offset 508. | <0.01% |
N11 | gnu-prefix-field | The prefix field of a header with GNU magic, which GNU tar uses for times in incremental archives. | none |
N12 | name-absolute | An absolute name, which points outside the directory the archive is extracted to. | <0.01% |
N13 | name-backslash | A backslash in a name: a character on Unix, a separator on Windows. | <0.01% |
N14 | name-bytes-after-nul | Bytes after the NUL that ends a name. | none |
N16 | name-dotdot | A name with .., which climbs out of the directory the archive is extracted to. | <0.01% |
N17 | name-empty | A member with an empty name. | none |
N18 | name-not-utf8 | A name that is not UTF-8. | <0.01% |
N19 | pax-nul-in-path | A NUL inside a pax path. | none |
N20 | pax-repeated-key | One pax key twice in one extended header. | <0.01% |
N21 | pax-global-path | A global pax header with a path. | none |
N22 | pax-path-empty | An empty pax path, which in POSIX deletes a path: the pax one, or the header's name too? | none |
N23 | extended-gnu-long-then-pax | A GNU long name, then a pax header, then the member: which of them does the long name apply to? | none |
N24 | pax-hdrcharset-binary | A pax path that is not UTF-8, declared as bytes with hdrcharset=BINARY. | none |
N25 | symlink-empty-target | A symlink with an empty target. | none |
N26 | pax-global-path-then-empty | A global path, then a global header that removes it with an empty value. | none |
N29 | extended-pax-path-before-gnu-long-name | A pax path, then a GNU long name, then the member: which name does it get? | none |
N30 | name-drive-letter | A name that starts with a drive letter, C:/drive-letter. | none |
N31 | pax-linkpath-not-utf8 | A pax linkpath that is not UTF-8. | none |
N32 | linkname-not-utf8 | A symlink whose target in the header is not UTF-8. | not measured |
N33 | type-solaris-extended | A Solaris extended header, typeflag X: a pax header, or a file? | none |
N34 | pax-linkpath-empty | An empty pax linkpath, which in POSIX deletes a target: the pax one, or the header's too? | none |
N36 | pax-gnu-sparse | A sparse file in GNU's pax format 1.0: the data as stored, or the file it expands to? | none |
N37 | extended-two-gnu-long-names | Two GNU long names for one member: the first, the last, or the header's? | none |
N38 | gnu-long-name-empty | An empty GNU long name: the name in the header, or none? | none |
N39 | gnu-long-name-size-without-nul | A GNU long name whose size leaves out its NUL, with bytes after it. | none |
N40 | pax-global-linkpath | A global pax header with a linkpath. | none |
N41 | pax-gnu-sparse-name | A pax GNU.sparse.name on a file that is not sparse, after a path. | none |
To reject: kinds¶
| identifier | name | what it is | prevalence |
|---|---|---|---|
F60 | trailing-slash-file-with-size | A regular file a/ with a size: a directory without data, or a file with data? | <0.01% |
K01 | kind-trailing-slash-file | A regular file whose name ends in a slash: a file, or a directory? | <0.01% |
K02 | linkname-on-file | A regular file with a link target. | <0.01% |
K03 | symlink-trailing-slash | A symlink whose name ends in a slash. | none |
K04 | name-empty-with-prefix | A regular file with an empty name and a prefix: dir/, or no name? | <0.01% |
K05 | hardlink-trailing-slash | A hardlink whose name ends in a slash: a link, or a directory? | none |
K06 | symlink-named-dot | A symlink .: dropped, renamed, or an error? | ≤0.08%* |
To reject: order and state¶
| identifier | name | what it is | prevalence |
|---|---|---|---|
S03 | hardlink-missing-target | A hardlink to a member that is not in the archive. | none |
S04 | hardlink-to-itself | A hardlink to itself. | 0.03% |
S05 | hardlink-to-later-member | A hardlink to a member that comes later. | none |
S06 | hardlink-to-symlink | A hardlink to a symlink: a link to the symlink, or to its target? | <0.01% |
S07 | tree-duplicate-file-differs | The same file twice, with different contents: the first copy, or the last? | not measured |
S08 | tree-duplicate-file-dir | A file, then a directory of the same name. | none |
S09 | tree-duplicate-file-identical | The same file twice, with the same contents. | 0.11% |
S10 | tree-symlink-then-file | A symlink, then a file of the same name: replace the link, or write through it? | none |
S11 | tree-through-symlink | A member written through a symlink, which points out of the destination. | none |
S12 | tree-under-file | A member under a file. | <0.01% |
S13 | hardlink-empty-target | A hardlink with an empty target. | none |
S14 | tree-symlink-then-dir | A symlink to a directory, then a directory of the same name. | none |
S15 | tree-file-then-hardlink | A file, then a hardlink of the same name to another file. | 0.03% |
S17 | hardlink-then-file-again | A file, a hardlink to it, then the file again: does the link keep the first contents? | 0.11% |
S18 | hardlink-absolute | A hardlink to /etc/passwd, in an archive with a member etc/passwd. | ≤0.03%* |
S19 | tree-through-symlink-inside | A member written through a symlink that points inside the destination. | none |
* An upper bound. The exact share is likely lower, and awaits a new scan of all tarballs.
Data to support the default policy¶
The default policy accepts 99.80% of the 82,586 distinct release tarballs of Spack, nixpkgs, Guix and live-bootstrap, and samples of PyPI, crates.io and RubyGems. See the list of the 168 it rejects, and why.
| ecosystem | tarballs | rejected | share |
|---|---|---|---|
| Spack | 37,553 | 58 | 0.15% |
| nixpkgs | 18,990 | 74 | 0.39% |
| Guix | 10,168 | 28 | 0.28% |
| PyPI | 10,000 | 17 | 0.17% |
| crates.io | 5,000 | 0 | none |
| RubyGems | 5,000 | 6 | 0.12% |
| live-bootstrap | 171 | 0 | none |
| total (distinct) | 82,586 | 168 | 0.20% |
The tarballs by the median mtime of their members, a rough check that the data spans enough years, leaving out the 14,361 whose mtimes are fixed for reproducibility or missing:
Motivating examples¶
| case | the archive | what implementations make of it |
|---|---|---|
| N09 | two pax headers for one file, with path=first and path=second | GNU tar, BusyBox, 7-Zip, node-tar, tar-fs, Go, R and Julia extract second; Python and Java extract first; PHP, RubyGems, Perl and live-bootstrap extract header-name, the name in the ustar header; bsdtar, Rust, .NET and Haskell reject it |
| S10 | a symlink a to target, then a file a | most replace the link with the file; Python's tarfile and live-bootstrap write through the link, to target; 7-Zip, .NET and Haskell reject it |
| K01 | a regular file named a/ | GNU tar, bsdtar, BusyBox, 7-Zip, node-tar, Java, Perl and live-bootstrap create a directory a; Python, Go, Rust's tar and astral-tokio-tar, PHP's PharData, RubyGems and R an empty file a; tar-fs, .NET, Haskell, Julia and tar-codec reject it |
| F12 | a size field of 000000 1000, with a space | GNU tar, BusyBox, 7-Zip, node-tar, tar-fs, Perl, Haskell, Julia and live-bootstrap read 0, and find a second file smuggled in the data; R reads 512, and extracts it as the data of a; bsdtar, Python, Go, Rust, Java, .NET, PHP and RubyGems reject it |
Contributing¶
This project is looking for contributions from people who work on package managers, registries and tar implementations, in the form of discussions in issues or pull requests on GitHub or by email to harmen at stoppels dot ch, so that the working draft of the default policy can become a reference.
You can also contribute by testing other implementations, or by crafting a tarball that is particularly hard to parse. A case that shows a vulnerability that is not yet disclosed is reported privately first: see SECURITY.md.
Contributors: