Sprocket | St. Jude Rust Labs

Sprocket. The Bioinformatics Workflow Engine.

Sprocket is a high-performance, modern, and open-source workflow engine for bioinformatics. Create, test, and run your analyses locally, then seamlessly move to HPC or the cloud to handle thousands of parallel workflows.

version 1.3

task say_hello {
    input {
        String greeting
    }

    command <<<
        echo "~{greeting}, world!"
    >>>

    output {
        String out = read_string(stdout())
    }

    requirements {
        container: "ubuntu:latest"
    }
}

$ sprocket run example.wdl --target say_hello greeting="Hello"
{
  "say_hello.out": "Hello, world!"
}

Check

Validate every document before it runs.

sprocket lint points to the exact line of each problem and suggests a fix. Invalid WDL exits with a non-zero code, so the same check can guard your CI, and sprocket format keeps every file in one style.

Read about check and lint

Terminal

$ sprocket lint example.wdl
note[ContainerUri]: container URI uses a mutable tag
   ┌─ example.wdl:18:20
   │
18 │         container: "ubuntu:latest"
   │                    ^^^^^^^^^^^^^^^
   │
   = fix: replace the mutable tag with its SHA256 equivalent

warning[UnusedInput]: unused input `color`
   ┌─ example.wdl:30:16
   │
30 │         String color = "green"
   │                ^^^^^

Edit
See the same checks while you write.
The Visual Studio Code extension runs sprocket analyzer, Sprocket's language server, and lists validation and lint problems as you edit. It also adds WDL syntax highlighting and snippets. For Neovim, use sprocket.nvim.
Set up the editor extension
Problems
example.wdl~/Desktop4 problems
- unused input `color`Sprocket(UnusedInput)[Ln 30, Col 16]
- task `say_hello` is missing both `meta` and `parameter_meta` sectionsSprocket(MetaSections)[Ln 3, Col 6]
  example.wdl[Ln 3, Col 6]: fix: add both the `meta` and `parameter_meta` sections
- container URI uses a mutable tagSprocket(ContainerUri)[Ln 18, Col 20]
  example.wdl[Ln 18, Col 20]: fix: replace the mutable tag with its SHA256 equivalent (e.g., `ubuntu@sha256:foobar` instead of `ubuntu:latest`)
- workflow `main` is missing both `meta` and `parameter_meta` sectionsSprocket(MetaSections)[Ln 22, Col 10]
  example.wdl[Ln 22, Col 10]: fix: add both the `meta` and `parameter_meta` sections
Run locally
Develop on your own machine first.
Write, run, and debug a workflow on your laptop before it goes anywhere near a cluster. sprocket inputs writes a template of the inputs a workflow expects, sprocket validate checks your values, and sprocket run executes it right there with the local Docker backend.
Take the guided tour
TerminalYour machine · Docker
```
$ sprocket run example.wdl --target main name="World"
{
  "main.messages": [
    "Hello, World!",
    "Hallo, World!",
    "Hej, World!"
  ]
}
```

Scale

Move to a cluster by changing configuration, not code.

Point sprocket.toml at a Slurm or LSF cluster with Apptainer, or at a Task Execution Service (TES) server. Sprocket moves inputs and outputs to and from cloud storage as the workflow needs them. It is built toward a target of 20,000+ concurrent jobs.

Configure a backend

[run.backends.default]
type = "docker"

[run.backends.default]
type = "slurm_apptainer"


default_slurm_partition.name = "gpu"
default_slurm_partition.max_cpu_per_task = 64
default_slurm_partition.max_memory_per_task = "96 GB"

The Slurm + Apptainer backend is experimental.

[run.backends.default]
type = "lsf_apptainer"


default_lsf_queue.name = "standard"
default_lsf_queue.max_cpu_per_task = 64
default_lsf_queue.max_memory_per_task = "96 GB"

The LSF + Apptainer backend is experimental.

[run.backends.default]
type = "tes"

url = "<tes-server-url>"

inputs = "<cloud-storage-url>"

outputs = "<cloud-storage-url>"

Reproduce

Every run leaves a complete record.

Each run gets its own directory with its inputs, outputs, logs, and every task attempt, and a SQLite database keeps the history. Turn on the call cache and a rerun after a failure reuses the tasks that already succeeded.

How provenance tracking works

./out

./out/
├── sprocket.db              # SQLite provenance database
├── runs/
│   └── main/
│       ├── <timestamp>/
│       │   ├── inputs.json
│       │   ├── outputs.json
│       │   ├── output.log
│       │   └── calls/        # One directory per task call
│       └── _latest -> <timestamp>/
└── index/                   # Optional output indexing

sprocket.toml

[run.task]
cache = "on"

Extend

Use our supporting tools, or build your own.

Lint every pull request with the Sprocket GitHub Action. To build your own tooling, analyze WDL from Python with sprocket-bio, use the wdl crate in Rust, or submit runs to the experimental Sprocket server over its REST API.

Start with the Python bindings

.github/workflows/wdl.yml

steps:

  - uses: stjude-rust-labs/sprocket-action@main
    with:
      action: lint
      except: TrailingComma,ContainerUri

$ pip install sprocket-bio


$ cargo add wdl

$ sprocket dev server start \
    --allowed-file-paths /home/user/workflows \
    --port 8080


$ curl -X POST http://localhost:8080/api/v1/runs \
    -H "Content-Type: application/json" \
    -d '{"source": "/home/user/workflows/hello.wdl", "inputs": {"name": "World"}}'

The Sprocket server is experimental and may change.

In the animation, Sprocket, a workflow engine built on WDL, is the head unit. Below it sits Crankshaft, a language-agnostic execution engine that handles the submission, execution, and reporting of tasks. Head units for other languages, such as CWL, Nextflow, and Snakemake, can drive it too.

Sprocket. The Bioinformatics Workflow Engine.

Check

Edit

Run locally

Scale

Reproduce

Extend