Measure the hardware capacity a rented container can actually use.
GPU concurrency · effective CPU cores · memory bandwidth · durable disk speed
Demo results · Methodology · Vast.ai runner · Documentation
Overview
Vast Benchmarking is a bounded Python benchmark for comparing Vast.ai Docker offers on computer-vision GPU throughput, effective CPU concurrency, single-thread CPU speed, memory throughput, and storage performance.
Each run produces portable JSON. The project can also store normalized results in SQLite and serve them through a Flask dashboard with category rankings and notes for individual machines.
Important
The standard profile has a hard 540-second wall-clock budget. Marketplace core counts and prices are context; rankings use capacity measured inside the container.
Measured demo
The demo snapshot has 19 stored runs from 15 accepted machines. In all, 60 GPUs finished the test. The leading results are:
- 118,420.80 images/s of concurrent GPU CV throughput
- 598.504 TFLOP/s of aggregate FP16 compute
- 368.64 effective CPU cores inside one rental
- 5.333 GB/s of single-thread SHA-256 throughput
See the full demo benchmark snapshot for machine IDs, exact results, historical rental rates, campaign cost, and acceptance caveats.
Workloads
| Area | What is measured |
|---|---|
| GPU | Concurrent FP16 GEMM, synthetic CV convolution, transport, utilization, power, and temperature |
| CPU | Effective cgroup/affinity capacity, single/all-core SHA-256, and Torch matmul |
| Memory | NumPy copy, fill, and scale bandwidth |
| Disk | Fsync-backed sequential writes and cache-advised reads |
| Dashboard | Percent-of-best rankings, historical price context, perf/$ sorting, and durable machine notes |
The detailed contracts and interpretation rules live in benchmark methodology.
Quick start
uv sync --extra dev --extra server uv run vast-benchmark run \ --profile smoke \ --db results/benchmarks.sqlite \ --output results/local-smoke.json uv run vast-benchmark serve \ --db results/benchmarks.sqlite \ --host 127.0.0.1 \ --port 8080
Open http://127.0.0.1:8080. Health is available at /healthz, and normalized data at
/api/runs.
Docker and Vast.ai
docker compose run --rm benchmark docker compose up dashboard
The controller can benchmark one offer or run a bounded batch. Before renting anything, it checks the hourly rate and worst-case spend. It connects over key-only SSH, rejects partial results, and destroys the exact instance it created.
Read running on Vast.ai before using paid infrastructure.
Note
Referral disclosure: Create a Vast.ai account with this referral link. If you sign up through it, the project maintainer may receive Vast.ai account credit. The referral has no effect on benchmark selection, rankings, methodology, or reported prices.
Safety model
- Provider credentials stay on the controller and are never copied to rented hosts.
- Partial GPU runs and superseded results remain inspectable but do not enter rankings.
- Rental prices are historical observations captured at run time, not current quotes.
- Result databases, raw artifacts, keys, and provider logs are excluded from Git.
See Security and the public-release checklist.
Documentation
| Guide | Purpose |
|---|---|
| Documentation index | Map of user, operator, methodology, and maintainer docs |
| Demo benchmark snapshot | Measured leaders, rates, costs, and caveats |
| Benchmark methodology | Workloads, effective cores, acceptance, and scoring |
| Running on Vast.ai | Single-offer and parallel campaign operation |
| Self-hosting | Generic Flask, Gunicorn, and reverse-proxy deployment |
| Contributing | Development and pull-request expectations |
Development
uv run ruff check src tests scripts uv run pytest node --check src/vast_benchmarking/static/app.js uv run python scripts/check_markdown_links.py uv run python scripts/check_public_release.py uv build uv run python scripts/check_distribution.py
Releases
Versioned wheels, source archives, and checksums are published through GitHub Releases. PyTorch remains an environment dependency supplied by the CUDA container.
License
Vast Benchmarking is available under the MIT License.