GitHub - Gysho/LLMrPro: LLM Router PRO: Your machines run the models. The cloud only fills the gaps. Call every model through one OpenAI format. Self-hosted, and yours to own

Your machines run the models. The cloud only fills the gaps. Call every model through one OpenAI format — self-hosted, and yours to own.

LLMrPro is a self-hosted LLM router. It exposes one OpenAI-compatible API and serves each request from a pool of machines you own — desktops, workstations, servers — running local inference engines. When no local worker can serve a request, it transparently falls back to the cloud provider you configured for that tier. One endpoint, one request format, your hardware first, your keys, your data.

What it does

• One OpenAI-compatible API for everything. Point any OpenAI client or SDK at your LLMrPro server — POST /{frontier,mini,nano}/v1/chat/completions with a standard Chat Completions body. Swap models and providers behind it without changing a line of caller code.
• Pools your own machines as inference workers. A lightweight desktop agent runs on each machine, connects outbound over a websocket (no inbound ports, no firewall changes), and pulls inference jobs. Each worker runs a full model on a local engine — LM Studio, Ollama, vLLM, llama.cpp, or the bundled MLX engine on macOS.
• Automatic per-tier cloud fallback. Three tiers — frontier, mini, nano — each with its own cloud backend (OpenAI, Anthropic, Google, Azure OpenAI). If the local pool can't serve a tier, the request falls back to that tier's cloud provider, with configurable retry/backoff.
• Single-tenant and self-hosted. One LLMrPro server per organization. Your keys, your data, your infrastructure — nothing leaves your control unless a request falls back to a cloud provider you configured.
• Live pool management. A built-in admin UI — device pairing, agent revocation, tier config, dispatch log, and health — backed by real-time pub/sub.
• Security built in. Per-source rate limiting, an egress guard (SSRF protection), provider keys encrypted at rest, signed agent tokens with expiry, and a checksum-verified install path.

How it works

   OpenAI-compatible          ┌────────────────────────────────┐
   client / app  ───────────▶ │        LLMrPro server          │
   (Bearer key)               │      (router + admin UI)       │
                              │  tier: frontier / mini / nano  │
                              └───────┬────────────────┬───────┘
                        websocket     │ dispatch        │ fallback
                        (job pull)    ▼ (local first)   ▼ (per tier)
                       ┌───────────────────────┐   ┌───────────────────┐
                       │  desktop agents        │   │  cloud providers  │
                       │  (your machines)       │   │  OpenAI/Anthropic │
                       │  MLX / Ollama / vLLM   │   │  / Google / Azure │
                       └───────────────────────┘   └───────────────────┘

1. A client sends a standard OpenAI Chat Completions request to a tier endpoint with the bearer key.
2. The server picks an eligible paired agent and dispatches the job; the agent runs it on its local engine and streams the result back.
3. If no agent can serve the tier, the server calls that tier's configured cloud provider and returns the result in the same OpenAI format. X-LLMrPro-* response headers report who served the request and why.

Why LLMrPro is different

Plenty of projects touch part of this; as of 2026 no single open-source project combines all of it.

The distinctive combination — federate a fleet of end-user desktops via a first-party agent + tiered cloud fallback + single-tenant self-hosting, behind one OpenAI-compatible API — is what LLMrPro is for.

The desktop agent is distributed unsigned, with SHA-256 verification at install time as the integrity mechanism. That is a deliberate tradeoff, not an oversight — see SECURITY.md.

Repository layout

LLMrPro/
├── balancer/     # the LLMrPro server: Meteor + MongoDB + React admin UI + deploy script
├── agent/        # the desktop worker: Electron + React app + installer template
├── .github/      # CI: unsigned agent builds for macOS + Linux
├── DESIGN.md     # full architecture, wire protocol, routing algorithm
├── SECURITY.md   # security model + how to report a vulnerability
└── CONTRIBUTING.md

On the balancer/ directory name. The project was called Balancer before it was renamed to LLMrPro. The directory, the Meteor method namespace (balancer.*), the collection prefixes and the X-Balancer-* response headers still carry the old name — they are wire and on-disk contracts that existing deployments and API consumers depend on, so they were deliberately left alone. Everything a user or operator actually sees says LLMrPro.

Quick start

LLMrPro has two installable pieces: the server (you run one) and the agent (installed on each worker machine).

1. The server

Requirements: Node.js 22+, MongoDB 6+ (a replica set is required for Meteor 3 change-stream reactivity), Meteor 3.4+, and — in production — a TLS-terminating reverse proxy (nginx / Caddy / Traefik).

Local / development

git clone https://github.com/gysho/LLMrPro.git
cd LLMrPro/balancer
meteor npm install
cp .env.production.example .env.production   # then fill in the secrets (see below)
npm run dev                                  # inject-secrets + meteor run on :3500

Open the admin UI at http://localhost:3500/admin.

Production — host prerequisites. LLMrPro makes no assumption about your host or cloud provider and ships no provisioning script — a single VM with the following is sufficient, and setting it up is yours to do (or to automate in your own infrastructure repo):

• Node.js 22+ and a MongoDB 6+ replica set
• A reverse proxy terminating TLS and forwarding to the app port (default 3500)
• A systemd unit — balancer.service by default — whose ExecStart runs node <remote-root>/current/bundle/main.js with ROOT_URL, MONGO_URL, MONGO_OPLOG_URL, METEOR_SETTINGS and HTTP_FORWARDED_COUNT set
• <remote-root>/settings.runtime.json holding the real secrets (never overwritten by deploys)

The exact environment and remote layout are documented at the top of balancer/scripts/deploy.sh.

Production — deploy a release (build → rsync → atomic symlink swap → restart):

cd balancer
./scripts/deploy.sh https://llmrpro.example.com deploy@your-host
curl -i https://llmrpro.example.com/healthz   # expect: 200 ok

Run ./scripts/deploy.sh --help for the full option list (--build-only, --restart-cmd, --remote-root, --keep-releases).

Operator CLI. balancer/ exposes an llmrpro bin for admin actions over DDP — pairing codes, agent listing and revocation, key rotation:

cd balancer && npx llmrpro --help

Secrets are never committed. settings.json holds ${VAR} placeholders; real values live in .env.production and are injected into settings.runtime.json at boot by inject-secrets.sh. You provide: a platform bearer key (stored only as a SHA-256 hash), an agent-token HMAC signing key, the bootstrap admin password, and one cloud API key per tier. The full set of variables is listed in balancer/.env.production.example.

2. The desktop agent (workers)

Each worker machine runs the agent, which connects outbound to the server (no inbound ports, no firewall changes) and forwards jobs to a local inference engine.

End users — one-line install (serves the right artifact for the OS/arch and verifies its SHA-256):

curl -fsSL https://llmrpro.example.com/install.sh | sh

Inspect it first with curl -fsSL https://llmrpro.example.com/install.sh.txt. On first launch the agent opens a Pairing screen — paste the server URL and a pairing code (an admin generates these in the admin UI), and the machine joins the pool.

Per platform — what the shipped build config actually produces:

• macOS 12+, Apple Silicon (arm64): .dmg, or the curl installer. Ships a bundled MLX inference engine; you can also point it at LM Studio / Ollama. Intel Macs are not a current build target.
• Linux x86_64: .AppImage, or the curl installer. Requires a local engine (LM Studio / Ollama / vLLM / llama.cpp).
• .deb / .rpm and Linux arm64 are supported by the installer and the download endpoint but are not built today — the electron-builder Linux target is AppImage x64 only. Enable them in agent/electron-builder.yml if you need them.
• Manual downloads are listed at https://llmrpro.example.com/agent/downloads, alongside the SHA-256 of each artifact.

Build, packaging and distribution details: agent/README.md.

Build the agent from source

cd agent
npm install
npm run build:linux      # .AppImage           (on a Linux host)
npm run build:mac        # .dmg + bundled MLX  (on a macOS host)

The desktop agent is distributed unsigned — no Apple Developer ID, no notarization, no Windows EV certificate. Integrity rests on HTTPS plus the SHA-256 the installer verifies before running anything. First launch on macOS therefore needs right-click → Open to get past Gatekeeper. The reasoning is in SECURITY.md.

Using it (API)

curl https://llmrpro.example.com/frontier/v1/chat/completions \
  -H "Authorization: Bearer <platform-key>" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-5",
        "messages": [{ "role": "user", "content": "Hello" }]
      }'

• Endpoints: POST /{frontier,mini,nano}/v1/chat/completions — the tier is the URL path.
• Body: standard OpenAI Chat Completions. Streaming, tools, and structured outputs pass through.
• Auth: Authorization: Bearer <platform-key> (required on every tier).
• The response is OpenAI-formatted whether served by a local agent or a cloud fallback.

Routing-provenance response headers. Every tier response reports who served it:

Every one of these is also emitted under the legacy X-Balancer- prefix (X-Balancer-Tier, and so on) for consumers written before the rename. Both families carry identical values and will continue to; prefer X-LLMrPro-* in new code. The optional request header X-LLMrPro-Budget-Ms — a caller-declared deadline, clamped to the operator ceiling — likewise accepts X-Balancer-Budget-Ms.

Configuration

All server config lives in balancer/settings.json (template: balancer/settings-example.json). Key blocks:

• publicHostname / urlSuffix — the public origin (and reverse-proxy path prefix, if any).
• tiers.{frontier,mini,nano}.cloudFallback — per-tier cloud backend: provider (openai / anthropic / google / azure), apiSurface, endpoint, apiKey, deployment, capabilities, parameterMap.
• rateLimits — completionsPerSecond, pairAttemptsPerMinutePerIp.
• trustProxy — set true behind a trusted reverse proxy to get per-client rate limiting.
• egressAllowlist — optional hostname allowlist for outbound cloud calls (SSRF hardening).
• cloudRetry, healthCheck, pairingCodeTtlMs, dispatchTimeoutMs, maxInflightPerAgent.

Secrets in settings.json are ${VAR} placeholders resolved from the environment at boot — never commit real values.

Documentation

• DESIGN.md — architecture, request flow, routing model, and the decisions behind them
• balancer/README.md — server layout, configuration, logging, running it locally
• agent/README.md — desktop agent: build, packaging, artefact distribution
• SECURITY.md — security model and how to report a vulnerability
• CONTRIBUTING.md — how to build, test, and submit a change

Security

The security model, the unsigned-agent distribution tradeoff, and how to report a vulnerability are in SECURITY.md. Provider keys are encrypted at rest, agent tokens are signed with expiry, and the server ships with rate limiting and SSRF egress protection.

Please do not open a public issue for a vulnerability — the disclosure process is in SECURITY.md.

Contributing

Contributions are welcome. CONTRIBUTING.md covers the dev setup, how to run the test suites, the conventions the codebase follows, and what a reviewable pull request looks like. Participation is governed by the Code of Conduct.

Quick version:

cd balancer && npm run test:unit    # 235 tests, no services required
cd agent    && npm test             # 108 tests

License

LLMrPro is released under the MIT License. © 2026 Gysho.