GitHub - jjba23/sift: Lisp, Scheme, Racket and Emacs Lisp formatter, linter and LSP (language server)

an idempotent formatter, a rule-based linter and a language server for Scheme, Racket, Emacs Lisp and Common Lisp

sift reads a Lisp source file, reports what is ill-formed in it, and rewrites its whitespace. The rewrite is a total function on text and a fixed point: formatting a formatted file returns it unchanged, and a comment, a blank line, a #; datum comment or a #. read-time form appears in the output in the place it was in the input. Both properties are asserted over every fixture in the suite, so neither is a promise that expires quietly.

Four dialects are read, each by the rules of its own report rather than by a compromise between them: Scheme, Racket, Emacs Lisp and Common Lisp. A square bracket is a vector in one of them and a clause in the next, sift does not average the two, and a Racket module is read from the #lang line that opens it rather than from the extension of the file it is in.

Install it with guix install sift or nix run, or clone the repository and run ./bin/sift version. The implementation is Guile 3.0 and the standard library, with no C dependency and no build step, because a Scheme program has none; sift doctor reports the interpreter it runs on, the library it loaded and the rules it found.

The same program is a language server (sift lsp). The findings, the fixes and the formatting are computed once and delivered either as text on standard output or as LSP responses, so Emacs through Eglot, Neovim and any other client of the protocol are told exactly what a pipeline is told.

It ships as one executable and one Emacs package, under the MIT license. The premise is the ordinary one: a Lisp program is a datum with a read syntax, and a formatter that treats it as such can be exact and cheap at the same time.

Check, format, lint. sift check prints the patch it would make and exits 1; sift format prints the formatted file; sift lint names the rule that fired, its severity and its position. The editor and the pipeline receive these same answers. Every line in that recording is real output, on files small enough to read in the picture.

Scope and audience

• Students of Lisp. Each dialect is read by its own report, so the tool in front of a learner enforces the conventions of the language being learned rather than a fourth one invented by a formatter.
• Guile and Guix code bases, whose cond clauses are written with square brackets and whose build files are gexps, want a formatter that agrees with the tree next door instead of re-deciding it.
• Emacs package authors who need the docstring, lexical-binding, interactive and provide rules in Flymake and in CI, from one program rather than three.
• Common Lisp projects in which package prefixes, LOOP keyword style and dangling parentheses are convention, and are preserved rather than normalised away.
• Polyglot Lisp repositories, which get one executable, one configuration file and one CI step instead of three of each.
• Editor and tool authors, who take the findings as data: --format=json, --format=sarif, GitHub annotations, or sift lsp.

Lossless rewriting is the whole of the argument, and the rest follows from it: a diff whose hunks are the change, a fix that replaces one range, a rule that can be silenced without editing the code it objects to.

Properties and guarantees

• Byte-exact preservation. Comments, blank lines, #; datum comments, #. read-time forms and #<...> objects return exactly as they were read. Formatting is a fixed point: the second run changes nothing. The suite asserts preservation and idempotence over every fixture.
• The dialect is determined, not assumed. Four dialects, one program: character literals, read macros, #| ... |# block comments, #; datum comments, package prefixes, LOOP and docstring conventions follow the dialect’s own report rather than a compromise. So does a square bracket, which is a vector in Emacs Lisp and a clause in Scheme, #{ ... }#, which is a symbol and only Guile reads, and a #lang line, which opens a Racket module and means nothing anywhere else. Guile is the dialect sift is written in and the one its reader knows most closely; the other three are read from their specifications.
• Three jobs, three commands. Formatting, linting and fixing are separate by design: format prints, --write writes, lint reports, lint --fix fixes, and ci asks the formatting question and the lint question in one pass without touching the tree.
• Ninety-five rules, each with its rationale. Every rule has a name, a stable code (SIFT012), a sentence, the dialects it applies to, an example and its own test file. sift explain RULE reads the reasoning out of the program, and doc/rules.org is generated from the rule records, so the book cannot drift from the code it describes.
• A fix that refuses to guess. A rule’s fix replaces exactly one range, and every fix of a run is discarded if together they would introduce a parse error. No delimiter is invented and no identifier is renamed.
• Suppression is a comment, and it stays where the code is. ;; sift:disable-next-line keyword-arguments switches one rule off for one line, disable for the rest of the form it sits in, and enable turns it back on. Suppression is line-oriented and form-scoped by design: an exception list in a configuration file is a list that no review ever reaches, and a range comment outlives the code it was written for.
• One language server, every editor. sift lsp speaks LSP over standard input and output: diagnostics, quickfixes and formatting, go to definition and references for all four dialects – including the names the standard library and its installed packages define, the documentation an implementation carries for the names it compiled in – written as a source file in that dialect, so a jump lands in code and not in a list of prose – and the compressed sources most Emacs installations ship – an outline, completion, inlay hints that name a call’s arguments, a hover that describes a symbol or the rule under it, import organization, a file rename that keeps its module in step, and a second opinion from the dialect’s own compiler when a buffer is saved. The standard library sift can index is the one of Scheme, Racket, Emacs Lisp and Common Lisp: Racket ships collects and pkgs as source, so its library is read like a project’s, and a name that only the compiled #%kernel defines is the one it cannot answer.
• Output for humans, editors and pipelines. The compile-style line that :make and compilation-mode parse, JSON for editors that want structure, SARIF 2.1.0 for code scanning, GitHub annotations for a pull request, and null for a script that cares only about the exit status.
• Configuration that can be checked in. YAML that is data and never code, four layers from system to command line, and sift config show --verbose reporting which layer won.
• Hooks, a doctor, and a catalogue that can be queried. sift install-hooks writes the pre-commit hook that runs ci; sift doctor reports which Guile it runs on, which library it loaded, how many rules it found and which configuration layer won; sift list-checkers prints the catalogue as a table and sift explain prints one rule’s reasoning, both read out of the rule records rather than out of a document that may have drifted from them.
• MIT licensed. A formatter belongs in as many pipelines as possible, and a licence question is a cost in the way of that with nothing to show for it.

One decision carries most of the list: a Lisp file is read as a tree of forms carrying source positions, not as lines of text, and the formatter prints that tree back. Preservation, a fix that replaces one range, and a diff whose hunks are the change all follow from it. That the reader and the printer are inverse operations is the oldest property Lisp has; the work here is to keep them inverse while the whitespace moves.

A first session

Guile 3.0 is the only requirement. The launcher inside a checkout runs the program, so a clone is enough:

git clone https://github.com/jjba23/sift.git
cd sift
./bin/sift version

A file that needs work:

$ cat square.scm
(define (square x)
(* x x))

$ sift check square.scm
--- square.scm
+++ square.scm
@@ -1,2 +1,2 @@
 (define (square x)
-(* x x))
+  (* x x))
$ echo $?
1

check returns 0 if the formatting is good. If not, it prints a patch to stdout and returns 1. It never modifies your files.

To make the change:

$ sift format --write square.scm
$ sift check square.scm; echo $?
0

A file with something to say:

$ cat small.scm
(define (go)
  (if (ready?)
      (proceed)))

$ sift lint small.scm
small.scm:2:4: warning: [if-without-else] `if` has no else branch; `when` says it once
1 file checked, 1 warning, 0 errors

Each line has the shape file:line:column: severity: [rule] sentence, which a compilation buffer, a CI log and grep already parse. Suggestions are findings too, and a deeply nested body is reported rather than rewritten:

$ sift lint messy.scm
messy.scm:5:3: info: [wrap-in-let] this body form nests 10 deep; bind part of it
messy.scm:6:5: info: [wrap-in-let] this body form nests 9 deep; bind part of it
messy.scm:7:7: info: [wrap-in-let] this body form nests 8 deep; bind part of it
messy.scm:8:9: info: [wrap-in-let] this body form nests 7 deep; bind part of it
1 file checked, 0 warnings, 0 errors

To read what a rule means, ask the program that holds it:

$ sift explain if-without-else
CODE
SIFT012

NAME
if-without-else

DESCRIPTION
`if` with no else branch, where `when` expresses the intent

DIALECTS
scheme

SEVERITY
warning

FAILING EXAMPLE
(if (ready?) (go))

EXPECTED FIX
(when (ready?) (go))

When the tool and the environment disagree about what is running, the program can be asked what it sees:

$ sift doctor
sift 1.31.0
  guile       3.0.11
  library     /home/joe/Hacking/sift/src
  rules       95
  dialects    scheme, racket, elisp, cl

Compilers
  scheme available
  elisp available
  cl not found

Sources
  cache       /home/joe/.cache/sift/sources
  downloaded  0
  tools       curl, tar

Configuration
  project  /home/joe/project/sift.yaml

Settings that are not the default
  brackets: square   # /home/joe/project/sift.yaml
  severity: warning   # /home/joe/project/sift.yaml

Try `sift ci .' to check a tree, or `sift explain RULE' to read a rule.

Two further commands follow from the same records. fix applies the fixes the rules can state as equivalences and reports what remains:

$ cat messy.scm
(define (f) (let ((x 1)) x))
(define (g)
  (if a b))

$ sift fix messy.scm
1 file checked, 0 warnings, 0 errors

$ cat messy.scm
(define (f) 1)
(define (g)
  (when a b))

It does not reformat, because formatting belongs to format, and a fix that moved whitespace would touch every line of the patch it was meant to make readable. It does tidy imports: the unused ones are removed and the specifications are sorted, both on by default, and optimize-imports or sort-imports in sift.yaml turns either off. ci is the same pair of questions asked by a pipeline, in one command and one status:

$ sift ci .
--- ./go.scm
+++ ./go.scm
@@ -1,3 +1,3 @@
 (define (go)
-    (if (ready?)
-        (proceed)))
+  (if (ready?)
+    (proceed)))
./go.scm:2:6: warning: [if-without-else] `if` has no else branch; `when` says it once
1 file checked, 1 warning, 0 errors
$ echo $?
1

Installation

At run time sift needs Guile 3.0 or newer and nothing else. Emacs is optional, and only for the editor half.

Requirements

The language server asks whichever of guild, emacs and sbcl is installed for its second opinion on a buffer; a compiler that is not present is simply not asked, and compiler-diagnostics: false turns the second opinion off.

sift runs on Linux and macOS. Caches follow the platform: $XDG_CACHE_HOME/sift when it is set, /Library/Caches/sift on macOS, and $XDG_CACHE_HOME/sift or /.cache/sift elsewhere. The optional tools above are found on PATH; a missing one is reported and the feature that needed it is skipped, rather than the tool refusing to run.

No C dependency and no build step are involved, and no package manager sits in the path: the program is Scheme source, loaded by whichever Guile is present.

From a checkout

git clone https://github.com/jjba23/sift.git
cd sift
./bin/sift version

bin/sift is a small POSIX shell launcher: it resolves its own symlink chain, puts the sibling src/ on Guile’s load path (or $SIFT_LOAD_PATH, if one is set), and hands the arguments to (sift cli). Where the launcher is not wanted, the program reduces to this:

#!/usr/bin/env guile
!#
(use-modules (sift cli))
(exit (sift-main (cdr (program-arguments))))

run with GUILE_LOAD_PATH pointing at src/. For a prefix, maak install copies the library to $(prefix)/share/guile/site/3.0 and the launcher to $(prefix)/bin; PREFIX=/usr/local is the default and maak uninstall is its inverse.

On the first run Guile compiles the library and prints a few ;;; note: ... lines to standard error while it does. That is Guile’s auto-compilation, not sift: it stops once the cache is warm, and GUILE_AUTO_COMPILE=0 suppresses it.

With Guix

From a checkout, the shell command builds the package it ships:

guix shell -f guix.scm -- sift version
guix build -f guix.scm        # or just the store path

To install it as a package, the guix-jointhefreeworld channel carries it:

The split between runtime and build inputs is deliberate. sift is a library plus a launcher, and needs only Guile. Veritas, the test framework, is needed to run the suite and is a native input of the package for that reason, so installing sift on a machine that will never run its tests installs no test framework.

With Nix

The repository carries its own derivation, so Nix does not need a second expression to build it:

nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix { }'
./result/bin/sift version

It also carries a flake: the same derivation with the ways in attached — a package, an app, a development shell and a container image:

nix run .#sift -- version        # in a checkout
nix run git+https://github.com/jjba23/sift -- version
nix build .#sift                 # ./result/bin/sift
nix develop                      # a shell with sift on PATH
nix build .#docker               # a loadable image, on Linux

The flake builds on Linux and on macOS (x86_64-darwin, aarch64-darwin), because the closure is Guile and Scheme and nothing else: on a Mac, nix run github:jjba23/sift is the shortest route to a formatter that runs. The image is the one output that is Linux only, and on Darwin it is absent rather than broken.

The nix-jointhefreeworld flake carries the same package, built from this repository’s own default.nix:

nix run git+https://github.com/jjba23/nix-jointhefreeworld#sift -- version

In a container

The image holds Debian’s Guile, the library and the launcher, and nothing else: there is no build step inside it, because a Scheme program has none. Build it from the checkout, or pull it:

docker build --tag sift .                  # or: podman build --tag sift .
docker run --rm --volume "$PWD:/src" sift check .

The container’s working directory is /src, so a volume mounted there is the tree sift reads; the entry point is sift itself, so the arguments after the image are sift’s arguments. A whole-project run, a patch, a lint and the language server (on standard input and output) all work the same way:

docker run --rm --volume "$PWD:/src" sift ci .
docker run --rm --interactive --volume "$PWD:/src" sift lsp

maak package-docker is the first command with the tags already on, and maak push-docker-latest / maak push-docker-tag publish it — the same three tasks maak’s own file has. The image is also a flake output for a machine with Nix and no engine: nix build .#docker writes a tarball that docker load < result loads.

A container exists for the other end of the pipe. On macOS no Guile is present that a script can rely on, and no Guix; Docker Desktop is. One image that runs, with a test over it, is cheaper than three install procedures that would each have to be verified on their own.

The Emacs package

emacs/sift.el is a single file with no dependencies beyond Emacs 28.1:

(add-to-list 'load-path "/path/to/sift/emacs")
(require 'sift)

The file carries the headers a package needs, so M-x package-install-file on emacs/sift.el also works, for a copy that should live in package-user-dir.

What is enabled after that is a matter for the project. The Emacs chapter has the two arrangements, with and without Eglot, and the settings that can be changed.

Verifying an installation

sift version   # the number, and that the program starts
sift doctor    # the Guile it runs on, the library it loaded, the rules it
               # has, and the configuration it decided on

The command line

Every command, every option and every exit code is in doc/cli.org, which is the reference. In brief: sift format, sift lint and sift check take files or standard input, sift ci is check and lint in one command for a pipeline, and the status is 0 for clean, 1 for findings and 2 for a usage error or a file that cannot be read.

How the dialect is determined

The determination is an ordered sequence of tests, and the first that answers wins:

1. --dialect, or the dialect setting.
2. The extension: .scm, .ss, .sld, .sls and .sps are Scheme; .rkt, .rktd and .rktl are Racket; .el is Emacs Lisp; .lisp, .cl, .asd, .lsp are Common Lisp.
3. The shebang: guile is Scheme, racket is Racket, sbcl, clisp, ecl and ccl are Common Lisp, emacs is Emacs Lisp.
4. A marker in the first 200 lines: a #lang line is Racket, (in-package ...) or (defpackage ...) is Common Lisp, (define-module ...) or (use-modules ...) is Scheme, a lexical-binding: t header is Emacs Lisp.
5. Otherwise it says so, and names the four dialects. Guessing is not among the answers: a tool that guesses at a dialect will one day rewrite a Common Lisp file as Scheme, and the diff will be blamed on the code.

An editor that has already resolved the language is better informed than any of this: the language server takes the identifier the client sends (scheme, racket, emacs-lisp, lisp) and goes by that.

Formatting

The formatter normalises line endings and spacing, places closing delimiters, computes indentation, aligns contiguous inline comments when asked, and preserves blank lines up to the limits of the style. Nothing else is touched: the position of a line break, the contents of a string and the spelling of an escape belong to the author. The function is deterministic and idempotent, and the suite formats every fixture twice to establish it. doc/formatting.org is the page: the two styles, the three indentation rules, every dialect’s read syntax, and the four things it will not do.

Linting

A rule is a record: a name, a code, a description, the dialects it applies to, a severity, a procedure and an example. The driver runs those the dialect selects, adds the reader’s own findings, applies the suppression comments, and emits the result in source order. The catalogue holds 95 of them, listed one by one in doc/rules.org. doc/linting.org covers the rest: what a fix may touch, why a fix is discarded rather than guessed at, and how a rule is silenced for a line or for a form.

The style, and the sources it is drawn from

The style is not an invention: it is the common practice of the four Lisp communities, written down where it can be argued with. doc/style.org is the long form – each choice, the alternative it beat and the reason – including which of Riastradh’s rules sift keeps and what it declines, the Emacs Lisp conventions that are really byte-compiler warnings, the reasons Common Lisp keeps its closers, the square bracket a Scheme cond clause is written with, and the rules sift declines to run on a Racket file because their premise is another dialect’s. The settings a project is likely to hold differently are listed in the configuration page: a project that has settled a convention should not have to argue with its formatter about it.

In Emacs

emacs/sift.el is a thin package: Emacs carries the text to the executable and the findings back, and every decision stays in one place. It requires Emacs 28.1 or newer and no external package. There are two ways in – Flymake for the rules, Eglot for the code actions – and doc/emacs.org is the page: the Flymake backend and its keymap, Eglot and the code actions, every setting in the customization group, and why sift and TAB can disagree about a third-party macro.

In other editors

Anything that speaks LSP can have the server, and anything that reads a compiler’s output can have the program. doc/editors.org carries the details and the recipes, and the ready-made ones live in the repository:

The human-readable format is file:line:column: severity: [rule] message, which Vim’s :make, Emacs’s compilation-mode and most CI log viewers already parse.

In the browser

web/ is the two jobs that are a function of a string — formatting and linting — compiled to WebAssembly with Guile Hoot, with an editor page around it. The same reader, the same formatter and all 95 rules run in the tab, and no input leaves it, because there is nowhere for it to go.

It is live at https://jointhefreeworld.org/sift/: an editor on a page, with the dialect, the style and the severity as controls, the findings in a panel that links back to the source, an outline, a completion popup, inlay hints that name a call’s arguments, and a rules browser built from the same catalogue the command line prints.

Why a compile, and not a port

The evident way to put a Lisp formatter in a browser is to write it again in JavaScript, and that is a second formatter, which is a second style: the two agree until the first file that exercises them, at which point the project has to decide which of its own tools it believes. The source of truth therefore stayed where it was. The tab runs the reader, the formatter, the lint driver and every rule that bin/sift runs, because they are the same modules.

The whole contract between the page and the Scheme is one procedure of one string to one string:

(web-request "{\"op\":\"lint\",\"source\":\"(if a b)\"}")
;; => "{\"ok\":true,\"result\":{\"warnings\":[…],\"counts\":{…}}}"

Everything the tab can do is that procedure: format, check, lint, fix, the rule catalogue, and the four editor questions (hover, go to definition, completion, outline), which are answered by the language server’s own analysis code against an empty workspace. Everything it cannot do requires a filesystem — no directory to walk, no sift.yaml, no save, no server — and the page reports nothing there rather than guessing.

What compiled, and what had to stand in

Hoot compiles R7RS and a subset of Guile, and the gap between that and what sift uses is web/compat/. Four Guile libraries have stand-ins: format, textual-ports, threads (a tab is one thread, so (sift pool) takes its sequential path) and ftw (scandir answers nothing, because there are no directories to walk — which is the truth about a tab and not a guess).

The demanding case was SRFI-13. Twenty-six files in the core import it, Hoot ships srfi-9, srfi-11 and srfi-14 and no SRFI-13, and Guile’s own srfi/srfi-13.scm is written with a re-export Hoot does not implement — so a browser build that reached for it would compile a re-export of names the target does not define and stop there. The answer is a stand-in of thirteen procedures, written against (scheme base), and a test that runs the browser’s copy in a second process and fails unless its format, lint and fix answers are byte for byte the answers Guile’s library gives. A stand-in that is never compared is a second implementation with no reason to agree with the first.

Four things could not be shimmed, and the source changed instead, in ways that leave a native build no worse off: (sift strings) carries the four string procedures R7RS leaves out, rather than importing a library the target may not have; three (char-set …) call sites became predicates, because a character set is an object R7RS does not have; one procedure takes its port in the body rather than as a keyword default that would be evaluated for nothing; and the filesystem question in (sift imports) asks whether this Scheme has stat and scandir rather than who is compiling it.

Nothing in src/sift/ names the browser, Wasm or Hoot. The dependency runs one way, and eight tests hold it that way: the core does not know that a tab exists; the facade’s dependency on the core is still five modules; every Guile library in the compiled closure has a stand-in; the SRFI-13 stand-in is the file a build finds; it answers what Guile’s own library answers, measured in a second process; the page names no URL from the root of a site; every URL it names carries the build stamp; and the editor asks for the wasm beside itself rather than at the root.

What it costs

The module is 3.4 MB, which is a formatter, a linter and 95 rules on a Wasm GC — a large page by the standards of a web page and a small one by the standards of an editor. Speed is the part of the experiment with numbers attached:

A screenful falls below the 300 ms debounce the editor applies to a lint, so the first two rows never reach the reader; past two kilobytes the cost is perceptible. There is no JIT, a record is allocated per node, and the reader allocates four records per character: that is the price of a Scheme on a Wasm GC, not something the facade can tune away. Where it matters, the request takes a checkers list, which is how a page asks for less work.

What the experiment says

Two questions were being asked. Whether a Scheme program of real size — not a toy, but a formatter with a reader of its own and 95 rules — can be compiled into something a tab runs: it can, with the caveats above, and the porting work amounted to one compatibility directory rather than a rewrite. Whether the seam survives contact: it does, and the test that establishes it runs under Guile in the ordinary suite, with no browser and no Wasm runtime involved.

The boundary is where the design is constrained. A tab has a buffer and no project, so two rules cannot fire and report nothing rather than guessing, and the language server is not in the tab at all: it is a process on standard input and output, which a browser is not. The experiment covers the formatter and the linter, and those are most of what a Lisp file needs while it is being written.

Building and publishing it

./web/build.sh     # writes web/dist/, and finds a Hoot that works
maak web-serve     # then open http://localhost:8080/
maak web-deploy    # or copy it to the directory nginx serves

The build is one command because Hoot is not a library sift ships: it is a compiler, needed once and never at run time, and locating it is the only difficulty. Hoot’s compiled modules are bytecode, and bytecode belongs to the Guile that compiled it, so guix shell guile guile-hoot yields a toolchain that fails halfway through a compile with an error raised inside Hoot; web/build.sh therefore compiles a two-line program to establish that its toolchain works, and asks Guix for guile-next guile-hoot when it does not. web/README.org has the long form of that, of the two things the server contributes (a .wasm content type and a trailing-slash redirect) and of the one it imposes (a two-day cache, which is what the build stamp in every URL is for).

Publishing is a copy into /srv/http/jointhefreeworld.org/sift on the machine that serves, which is how the rest of that site is published: no ssh, no rsync, no server to reload, because a file that is present is a file that is served.

What the language server answers

The extent of what sift lsp answers – request by request, including the requests it refuses, its code actions and the two limits that accompany them – is in doc/language-server.org. The limits bear repeating: a reference is matched by name within one dialect, because sift expands no macro and follows no import; and the project is read in the background, so a jump answers from the files read so far instead of waiting for the rest. A client that advertises experimental is told how far the reading has got by the sift/indexStatus notification (uninitialized, scanning, indexing, ready); the states, the bounds (100 ms for a jump, 200 ms for a reference search) and the batch size are in While the project is being read.

In a pipeline

One command asks both questions:

It checks the formatting, then lints, prints both reports in the format it was asked for, and exits 1 if either found something, 2 if a file could not be read or parsed. It never writes and never fixes: a job that edits the tree it is judging is a job whose verdict means nothing.

GitHub Actions

There is a reusable workflow, so most repositories need one file and no flags:

# .github/workflows/sift.yml
name: sift
on: [push, pull_request]
jobs:
  sift:
    uses: jjba23/sift/.github/workflows/sift.yml@v1.31.0

It annotates the diff with what it found and uploads the SARIF document to code scanning, so a finding that was not fixed becomes an alert in the Security tab rather than a line in a log that is never scrolled again. It needs no Guix and no build: the job installs Guile 3.0 from the runner’s packages and runs the checkout as it stands, and with: install: guix builds it instead for a caller that wants a store path. The same repository publishes a composite action, for a job that wants sift as one step among its own:

- uses: jjba23/sift@v1.31.0
  with:
    paths: src/ tests/
    severity: warning

Spelling the step out is also possible: --format=github writes workflow commands instead of prose, so each finding lands as an annotation on the line of the diff:

- name: sift
  run: ./bin/sift ci --format=github src/

::warning file=src/foo.scm,line=3,col=3,title=SIFT012 [if-without-else]::`if` has no else branch; `when` says it once
::error file=src/bar.scm,line=1,title=sift format::src/bar.scm would be reformatted by `sift format --write`

And --format=sarif writes one document for the whole run, which is the shape a code scanning service consumes:

- run: ./bin/sift ci --format=sarif src/ > sift.sarif
- uses: github/codeql-action/upload-sarif@v3
  if: always()
  with:
    sarif_file: sift.sarif

A pre-commit hook

sift install-hooks            # writes .git/hooks/pre-commit, running `sift ci'
sift install-hooks --force    # replaces one that is already there

The hook runs whichever sift is on PATH; set SIFT in the hook’s environment to name another copy. The pre-commit framework is also supported: .pre-commit-hooks.yaml declares three hooks — sift-format (rewrites), sift-check (fails) and sift-lint — so a project selects the one it needs:

repos:
  - repo: https://github.com/jjba23/sift
    rev: v1.31.0
    hooks: [{id: sift-format}, {id: sift-lint}]

Parallel evaluation

A run over a project is a sequence of runs over single files, and the files are independent, so lint, format, check and ci distribute them over the processors and print the answers in file order: a parallel run writes exactly the bytes a sequential run writes. SIFT_JOBS is the only setting involved, and the two runs that are never parallel are the ones that write the files they read. The setting, the two exceptions and the reason the output is ordered are in doc/configuration.org.

Configuration

A project’s settings are a sift.yaml, or a sift.scm the project has trusted, read in four layers and validated against one schema: a key that sift does not have is refused with its position rather than ignored, and sift config show --verbose reports which layer won. doc/configuration.org has the whole of it – the discovery rules, the file-filter, the trust model, and the settings the language server reads out of the same files.

Testing

Every part of the project is tested with the framework its own users already have – Veritas for the library, ERT for the Emacs package, Vim and Node for the editor integrations – and none of the suite is a smoke test. doc/development.org is the page: the maak tasks, what the benchmarks measure, and what the suite actually asserts. Those tasks are also the pipeline: a pull request against trunk runs every gate before the branch lands, and only the push that lands it publishes the browser build (the pipeline).

Architecture

src/sift/ is one module per job, and the dependencies run one way: records is a vocabulary, reader and format use it, the checkers use records and check/base, lint assembles them, render prints their findings, lsp delivers the same findings over a protocol, and cli wires the lot to the command line. Nothing in format or in a checker knows about ports or exit codes. The module map, and what each module owns, is doc/development.org.

Premises

Six premises the code is measured against: a source file is a datum, not a string; a guarantee that is not a test expires quietly; the author decides what the code says and the tool decides where the whitespace goes; a dialect is a language, not a variant; state what can be decided and report no finding otherwise; and everything a program says should be available as data. doc/design.org states them, and then the decisions that follow from them – a reader of its own, one walk with the answers kept, fuzzy formatting rather than pretty-printing, and a YAML reader small enough to audit.

Limits

sift itself does not expand macros, does not type-check, and does not know a project’s conventions. It reports; the author decides. That is not modesty – a linter that decided would be useless on the first project it had never seen. Where the answer can be wrong, the rule says so: wrap-in-let is a suggestion, keyword-arguments counts, and the thresholds are settings. The language server can ask the project’s own compiler for the checking sift does not do – on open and on save, in a thread, on a temporary copy of the buffer – and publishes what that compiler says about the names, the arity and the types it can see; compiler-diagnostics: false turns the second opinion off. The language server answers the requests listed on the language server page, and answers the others with “method not found” rather than a guess.

Contributing

The list of Lisp dialects that deserve a better reader is long, and each item on it is a contribution that can be carried to the end:

• A bug report. For a formatter the informative report is the input, the output that was expected, and sift version with the first section of sift doctor – the dialect chosen and the rules that ran usually answer the question on their own. A file that sift refuses to parse is a bug even when the file is broken: the message is obliged to say where.
• A rule. Every rule is one record and one test file, and the catalogue, the documentation and the editor integrations follow from them. A rule needs a name, a code (the next free one), a sentence, the dialects it applies to, a severity, an example, and the three cases – one that passes, one that fails at a known line and column, and, when the fix can be stated safely, the fixed text and a re-lint that must find nothing.
• A dialect detail. An escape spelled differently, a comment syntax in an unexpected place, a form whose body indents by something other than two: tests/reader-scheme.scm and tests/format-scheme.scm are where that behaviour is written down, and the three Racket suites – reader, formatter and linter – are the same way of saying it for the fourth dialect.
• An editor integration. Neovim, Vim and VS Code each have a directory and a test written in that editor’s own tool. Another editor is one directory more.
• A word in the documentation. The README, the style document and the rule descriptions are written by hand and read by people deciding whether to trust the tool. A sentence that causes hesitation is a defect in the documentation.

A first patch, in order

1. Read doc/style.org: it is the argument the rules implement, and a rule that disagrees with it is the informative kind of bug report.
2. Run the suite once before changing anything, so that a failure afterwards is one the patch is responsible for: maak test (and maak emacs-test if the package was touched).
3. Add the rule or the fix with its test file under tests/checkers/, then run maak self-check, maak check and maak fixture-check.
4. Send it as a pull request, or as a patch – see below.

The project’s own style is the style it enforces, so it formats itself:

maak fmt           # format sift's own sources with sift
maak test          # the Veritas suite
maak check         # compile every module
maak self-check    # sift's sources are formatted by sift
maak fixture-check # the golden corpus still round-trips
maak docs          # regenerate doc/rules.org
maak emacs-check   # byte-compile the package, warnings are errors
maak emacs-test    # ERT
maak editors-check # Vim in batch, Node over the VS Code logic

Docstrings are Texinfo-flavoured (@var{}, @code{}), the first line is a sentence, and each paragraph states the reason as well as the rule – which is why this README can be read without a specification beside it.

A commit message is a type and then a sentence: feat: for something that did not exist before, fix: for something that was wrong, and docs:, test:, refactor:, build:, ci: or chore: for the rest, scoped in parentheses when that narrows the subject (fix(lsp):, refactor(reader):). The reader who matters is the one scanning git log for the change that broke a build, and the type is what lets nine commits in ten be skipped unopened.

Patches arrive as a pull request on GitHub or as a patch sent with git format-patch and git send-email, which is how the author usually works. The review concerns the code.

Questions are equally welcome as issues. A question that the README does not answer is a defect in the README, and reporting it is the most useful bug report there is.

Further reading

In this repository

• doc/style.org — the style, the choices behind it, why each one is idiomatic Lisp, and which of Riastradh’s rules sift keeps.
• doc/rules.org — every rule, with its code, in a document generated from the rules themselves.
• doc/editors.org — Emacs, Eglot, Neovim, Vim, VS Code, Helix, pre-commit, and the two CI shapes.
• doc/cli.org — every command, every option, and what each exit status means.
• doc/language-server.org — every request sift lsp answers, the ones it refuses, and the code actions it offers.
• doc/diagrams.md — the moving parts as Mermaid diagrams: navigation, the two diagnostic passes, standard-library discovery and downloads, import organization, and file renames.
• doc/configuration.org — the two configuration files, the four layers they are read in, the settings a project is likely to change, and the one knob that is not a setting.
• doc/formatting.org — the formatter: the two styles, the three indentation rules, and every dialect’s read syntax.
• doc/linting.org — the lint driver, what a fix may touch, and how a rule is silenced.
• doc/emacs.org — the Emacs package, both ways in, and every setting.
• doc/design.org — the convictions, and the decisions they produced.
• doc/development.org — the maak gates, the pipeline that runs them and the ritual that cuts a release, what the benchmarks measure, and the module map.
• web/README.org — the browser build: the seam, the protocol, the compatibility layer, and the numbers.
• sift explain RULE — the same reasoning, one rule at a time, from the program.
• sift list-checkers — the catalogue as a table, filtered by dialect.
• emacs/sift.el — the package, with its commentary at the top.

Outside it

• Riastradh’s Lisp Style Rules — Taylor R. Campbell’s guide, and the most compact argument for the style sift enforces; which of its rules sift keeps is in the style page.
• Google’s Common Lisp style guide — the other modern reference, and the source of the community style sift applies to Common Lisp.
• R7RS-small — the report against which Scheme is read, and the reason the compatibility layer is written in the subset it is.
• SRFI 13 — the string library the command line takes from Guile and a browser build provides through thirteen procedures of its own.
• The Emacs Lisp manual’s docstring tips — the conventions behind the Emacs Lisp rules, which record what a byte-compiler reports rather than a matter of taste.
• The language server protocol — the specification sift lsp implements, and the source of the refusals it is allowed to make.
• Guile Hoot — the compiler behind the browser build.

License

sift is free software under the MIT license; LICENSE carries the text. A short permissive license is a practical requirement rather than a sentiment: a formatter that spreads through other people’s pipelines cannot carry a licensing question with it.