Pointing at the error: compiler-style diagnostics in uutils coreutils

Pointing at the error: compiler-style diagnostics in uutils coreutils

For 50 years, Coreutils have never stopped evolving. Now, we're pushing that innovation further by rethinking how they report errors.

Unix tools report errors as a single line on stderr. That line says what went wrong, not where. For most commands there is nowhere else to point anyway, but a few take arguments that are small languages: a test expression, a chmod mode, a sort key, a tr set. When one of those fails to parse, what you actually want to know is which argument, or which character of it, the parser tripped over.

rustc has been answering that question with a caret for years, and ariadne puts the same rendering one dependency away. The idea of bringing it to a command-line tool comes from uutils awk, which already reports errors in an awk program that way; coreutils arguments are smaller languages, but they parse just the same. Starting with 0.11.0, coreutils uses it. When stderr is a terminal, a parse error is printed as a report: the arguments are echoed back as a source line, a caret marks the culprit, and a help line explains the syntax when we have something useful to say about it.

What it looks like

Start with tr, whose GNU message assumes you already know what a collating sequence is.

Before:

tr: range-endpoints of 'y-b' are in reverse collating sequence order

After:

tr: range-endpoints of 'y-b' are in reverse collating sequence order
   ╭─[ tr:1:7 ]
   │
 1 │ tr qw[y-b] x
   │       ─┬─
   │        ╰─── did you mean 'b-y'?
   │
   │ Help: a range goes from the lower character to the higher one, as in a-z
───╯

Try it in the playground.

A cut list can be long, with a single bad item in it.

Before:

cut: invalid decreasing range
Try 'cut --help' for more information.

After:

cut: invalid decreasing range
   ╭─[ cut:1:10 ]
   │
 1 │ cut -f 1,4-2,9-12 notes.txt
   │          ─┬─
   │           ╰─── this range ends before it starts
   │
   │ Help: a list is N, N-M, N- or -M, separated by commas, as in -f1,4-6,9-
───╯
Try 'cut --help' for more information.

Try it in the playground.

The caret does not have to cover a whole argument. It can land on one character.

Before:

chmod: invalid operator (expected +, -, or =, but found ?)

After:

chmod: invalid operator (expected +, -, or =, but found ?)
   ╭─[ chmod:1:5 ]
   │
 1 │ g+rw?x notes.txt
   │     ─
   │
   │ Help: a mode is either octal, as in 644, or clauses such as u+rwx,go-w
───╯

sort keys are short enough that a stray character is easy to miss.

Before:

sort: stray character in field spec: invalid field specification '2.3x'

After:

sort: stray character in field spec: invalid field specification '2.3x'
   ╭─[ sort:1:11 ]
   │
 1 │ sort -k2.3x notes.txt
   │           ─
   │
   │ Help: a key is FIELD[.CHAR][OPTS][,FIELD[.CHAR][OPTS]], as in -k2.3,4nr
───╯

Try it in the playground.

env -S takes a whole command line and splits it the way a shell would. The old message could only quote the offending fragment back at you. Note that the string contains spaces, so it is echoed back quoted, and the caret still lands inside the quotes.

Before:

env: only ${VARNAME} expansion is supported, error at: ${1FOO}

After:

env: only ${VARNAME} expansion is supported, error at: ${1FOO}
   ╭─[ env:1:14 ]
   │
 1 │ env -S 'echo ${1FOO}'
   │              ─┬─
   │               ╰─── a variable name cannot start with a digit
   │
   │ Help: only $NAME and ${NAME} are expanded; the other shell forms are not
───╯

test builds its expression out of separate arguments. The report echoes the expression on its own, without the test in front, and marks the argument that broke it.

Before:

test: invalid integer 'zap'

After:

test: invalid integer 'zap'
   ╭─[ test:1:7 ]
   │
 1 │ 7 -eq zap
   │       ───
   │
   │ Help: -eq, -ne, -lt, -le, -gt and -ge compare integers; use =, !=, < or > to compare strings
   │       -eq equal, -ne not equal, -lt less than, -le less than or equal, -gt greater than, -ge greater than or equal
───╯

Try it in the playground.

A SIZE is a number followed by a unit. The report says which half was rejected.

Before:

head: invalid number of bytes: '1fb'

After:

head: invalid number of bytes: '1fb'
   ╭─[ head:1:10 ]
   │
 1 │ head -c 1fb notes.txt
   │          ─┬
   │           ╰── not a known unit
   │
   │ Help: a size is a number and an optional unit: K, M, G and so on for 1024, KB, MB, GB for 1000
───╯

Try it in the playground.

One parser handles every SIZE in the suite, so the same report shows up for tail -c, truncate -s, split -b, shred -s, od -N, sort -S, the block sizes of du -B, df -B and ls --block-size, and the threshold of du -t.

numfmt --format is a printf-style format that allows exactly one conversion. The old message just restated the rule. The annotation names the conversion you actually wrote.

Before:

numfmt: invalid format '%q', directive must be %[0]['][-][N][.][N]f

After:

numfmt: invalid format '%q', directive must be %[0]['][-][N][.][N]f
   ╭─[ numfmt:1:18 ]
   │
 1 │ numfmt --format=%q 1000
   │                  ┬
   │                  ╰── f is the only conversion numfmt has; %d, %e, %g and the other C conversions are not accepted
   │
   │ Help: a format is [PREFIX]%[0]['][-][WIDTH][.PRECISION]f[SUFFIX], as in "%'-10.2f"
───╯

Try it in the playground.

csplit patterns contain regexes, and the regex engine already knows which character it choked on. We were simply throwing that position away.

Before:

csplit: '/a{2,1}/': invalid pattern

After:

csplit: '/a{2,1}/': invalid pattern
   ╭─[ csplit:1:20 ]
   │
 1 │ csplit notes.txt /a{2,1}/
   │                    ──┬──
   │                      ╰──── invalid repetition count range, the start must be <= the end
   │
   │ Help: a pattern is a line number N, /REGEXP/[OFFSET] or %REGEXP%[OFFSET], each optionally followed by {N} or {*}
───╯

Try it in the playground.

That label comes straight from the regex engine and is not translated, since it is the only place the wording exists.

Where it applies

28 utilities use it in 0.11.0. The linked examples run in the playground; the others are for utilities the WebAssembly build does not ship, so try those locally:

Compatibility first

Being a drop-in replacement for GNU coreutils comes first, so this is strictly an interactive nicety:

• Reports are only rendered when stderr is a terminal. In a script, a pipe or a test suite, each utility keeps printing exactly the plain one-line message shown as "Before" in the examples above, so anything that greps stderr keeps working.
• Exit codes are unchanged.
• Colors are only used on a terminal and respect NO_COLOR.
• Like the rest of uutils, the messages, labels and help lines are localized; translations are managed on Weblate.
• It can be compiled out to save a little space: the rendering sits behind the feat_diagnostics cargo feature (on by default), and building without it drops the ariadne dependency while every utility keeps its plain messages.

Turning it on and off

By default the rendering keys off stderr being a terminal and nothing else, which is usually but not always what you want. UUTILS_DIAG overrides it: always draws the report even into a file or a pipe, never keeps the plain line even at a terminal, and auto, or an unset variable, decides from stderr as before. An unrecognized value is deliberately not an error. This is the kind of variable people export from a shell profile once and forget about, and a typo in it should not be able to make a utility fail.

There is no command-line flag to go with it. The utilities that would need one most cannot have it: in test, printf and expr, a new option would be either illegal or ambiguous with the operands themselves.

To get a report out of a script or a CI log, to paste into a bug report for instance:

$ UUTILS_DIAG=always sort -k2.3x notes.txt 2> parse.log
$ cat parse.log
sort: stray character in field spec: invalid field specification '2.3x'
   ╭─[ sort:1:11 ]
   │
 1 │ sort -k2.3x notes.txt
   │           ─
   │
   │ Help: a key is FIELD[.CHAR][OPTS][,FIELD[.CHAR][OPTS]], as in -k2.3,4nr
───╯

Colors are decided separately, and still by the terminal: a report forced into a file is written without them, so there are no escape sequences to strip back out. NO_COLOR sits in between at a terminal, where the report is still drawn, just in plain text.

The two tricks people used before the variable existed still work. Sending stderr somewhere that is not a terminal gets the plain line:

$ sort -k2.3x notes.txt 2>&1 | cat
sort: stray character in field spec: invalid field specification '2.3x'

And giving a command a pty (script -qec "sort -k2.3x notes.txt" /dev/null, or unbuffer from expect) gets the report back, which is handy when the command has to run under a terminal for other reasons.

What comes next

coreutils is where this starts, not where it stops. The rendering lives in uucore::diagnostics, which the other uutils projects already depend on, so picking it up is mostly a matter of handing the parser's error a span. findutils and sed are being wired up now; a find expression and a sed script are exactly the kind of small languages a caret helps with, and grep, awk and the rest have the same regexes and format strings to point at.

If a utility you use still prints an unhelpful one-liner, patches are welcome!