GitHub - ndom91/llama-dash: The operations layer for your local LLM stack

llama-dash turns a self-hosted local inference box into an observable, policy-controlled AI gateway: one UI for model state, request history, API keys, routing rules, proxy metrics, and client setup. The implemented inference backend is currently llama-swap over llama.cpp.

It is the single public entrypoint for OpenAI-compatible and Anthropic-compatible clients. llama-dash owns proxy policy, logging, auth, routing, and backend normalization, your selected inference backend owns local model processes and inference when traffic is routed to local models.

OpenAI SDK / Claude Code / Continue / Open WebUI
                    │
                    ▼
              llama-dash :3000
      dashboard · auth · logs · routing · metrics
             │                     │
             ▼                     ▼
      llama-swap :8080         direct /v1 upstreams
  llama.cpp models · peers      OpenAI · Anthropic

✨ What it does

Watch the box — live request, token, model, upstream, GPU, update status, and static inference-host inventory in one dashboard.
Manage models — load/unload models, inspect per-model stats, capabilities, and llama-swap selector routing, view residency history, and edit llama-swap config with validation.
Proxy clients — expose one OpenAI/Anthropic-compatible /v1/* endpoint for local models, peers, direct upstreams, Claude Code, Continue, Open WebUI, and more.
Track requests — searchable request history with filters, histograms, detail views, attribution metadata, token counts, and cost estimates.
Control access — dashboard login, hashed API keys, per-key RPM/TPM limits, model allow-lists, MCP relay allow-lists, and per-key usage breakdowns.
Enforce policy — routing rules for model rewrites, rejects, passthrough auth, direct HTTPS upstreams, encrypted credentials, system prompts, and global request size limits.
Test models — playgrounds for chat, image, speech, and transcription, including article-to-speech extraction.
Export ops data — raw log streams, retention controls, request auditing, and low-cardinality Prometheus metrics at /metrics.
Move fast — g-leader keyboard navigation (g d dashboard, g r requests, g l logs, …), j/k/Enter through the request list, and H/L between request details.

_{Dashboard Live traffic, tokens, model residency, upstream and GPU health}	_{Playground Chat against local endpoints with request/response inspection}	_{Request detail Routing, attribution, latency, tokens, and payload metadata}	_{Logs Raw llama-swap, proxy, and upstream streams in one viewer}

_{Model detail Load history, stats, recent requests, and config context}	_{Speech playground Read any article and audio testing}	_{Policies Aliases, routing rules, passthrough auth, and request limits}	_{Requests Searchable history with filters, sorting, and histogram}

⚡ Quick start (Docker Compose)

Choose the compose file that matches your GPU vendor. Both setups use ./config/config.yaml for llama-swap config, ./models/ for model files, and expose llama-dash on http://localhost:3000.

First create your env and config files, then set the required secrets:

cp .env.example .env       # then set BETTER_AUTH_SECRET and CREDENTIAL_ENCRYPTION_KEY
cp config/config.example.yaml config/config.yaml   # edit models

The compose files load .env via env_file and won't start without it. Generate a session secret with openssl rand -base64 33. See the Environment table for all values.

AMD / ROCm

docker compose -f docker-compose.amd.yaml up -d

docker-compose.amd.yaml runs ghcr.io/mostlygeek/llama-swap:rocm, passes through /dev/kfd and /dev/dri, and also mounts /dev/dri into llama-dash so AMD GPU stats work in the dashboard. The config directory is mounted into both services so llama-dash can atomically save config.yaml and llama-swap can reload it through -watch-config.

NVIDIA / CUDA

docker compose -f docker-compose.nvidia.yaml up -d

docker-compose.nvidia.yaml runs ghcr.io/mostlygeek/llama-swap:cuda and requests gpus: all for the llama-swap service. This requires the NVIDIA Container Toolkit on the host. The config directory is mounted into both services so llama-dash can atomically save config.yaml and llama-swap can reload it through -watch-config.

🏗️ Manual setup

Requirements

Node 24+
pnpm
A reachable llama-swap instance

Install

cp .env.example .env   # edit INFERENCE_BASE_URL to point at your instance
pnpm install
pnpm db:migrate        # creates data/dash.db
pnpm dev               # http://localhost:5173

🏔️ Environment

Copy .env.example to .env and fill in the values.

See the environment variables reference for the complete list of variables, defaults, and deployment notes.

✴️ Claude Code / Anthropic passthrough

Route any Anthropic SDK, including Claude Code, through llama-dash for logging, filtering, per-request inspection, and Anthropic subscription passthrough.

See the Claude Code & Anthropic passthrough guide for client setup, routing rules, subscription OAuth, and provider-key flows. For remote MCP servers, see the MCP relays guide.

🤖 Acknowledgements

This project was developed with significant assistance from LLMs. Architecture decisions, implementation, and documentation were all shaped through human-AI collaboration.

📝 License

MIT