GitHub - shivnathtathe/UnlimitedNIM: Make NVIDIA's 40 RPM free tier feel unlimited. Smart priority queue proxy for coding agents.

UnlimitedNIM

Make NVIDIA's 40 RPM free tier feel unlimited. Smart priority queue proxy for coding agents.

Works with Cline · Roo Code · OpenCode · Any OpenAI-compatible agent

The Problem

NVIDIA's free NIM tier limits you to 40 requests per minute. Coding agents like Cline, Roo Code, and OpenCode generate bursts of traffic — an agent loop firing tool calls, context reads, and model invocations in parallel will slam straight into that wall. The result: a flood of 429 Rate Limit Exceeded errors, retried work, frustrated agents, and stalls in the middle of a task.

You don't get more throughput from NVIDIA — but you can stop wasting the window you have.

What This Is

UnlimitedNIM is a thin, streaming reverse proxy that sits between your coding agent and NVIDIA's API. It enforces NVIDIA's 40 RPM limit for you — in a way that's invisible to your agent:

• Sliding-window rate limiter — never fires more than 40 requests per minute, so NVIDIA never sees a burst and never returns 429.
• Priority queue — when the window is full, requests wait in line instead of failing:
  • priority 1 — user messages (you typed it, you're waiting)
  • priority 3 — agent loop tool calls (the bulk of traffic)
  • priority 5 — debug dumps and huge stack traces (fire when there's room)
• Full-duplex streaming — SSE responses stream straight through, chunk by chunk.
• Starvation protection — a low-priority request stuck longer than the TTL is promoted so it eventually fires.
• Backpressure — a bounded queue rejects with 503 + Retry-After instead of leaking memory or flooding NVIDIA.

The Screenshot-Worthy Stat

260+ real NVIDIA requests driven through the proxy — zero leaked 429s, zero rejections.

Wave 1 (40 concurrent) fired instantly. Wave 2 (60 more, fired while wave 1 was still in flight) queued and streamed in across the next window — every single request completed, and not one rate-limit error reached a client.

How It Works

Coding agent ──POST /v1/chat/completions──▶ UnlimitedNIM ──▶ NVIDIA API (40 RPM cap)
                      ▲                          │
                      └──── 200 + SSE stream ◄────┘
                        (or waits in priority queue)

1. A request arrives with an X-Priority header (1/3/5).
2. If the current 60s window has room, it fires immediately.
3. Otherwise it enters the min-heap priority queue — highest priority (lowest number) leaves first; same priority is FIFO.
4. As the window rolls over (or requests finish), the queue drains up to 40/min, never exceeding the limit.
5. Streaming responses relay byte-for-byte; client disconnects abort the upstream call and free the slot (the window slot is never refunded — NVIDIA already counted it, so we never re-fire).

Quick Start

# 1. Set your NVIDIA key
#    (either export it or copy .env.example → .env and fill it in)
export NVIDIA_API_KEY=your-key-here

# 2. Install
pip install -r requirements.txt

# 3. Run
python main.py
# INFO: Uvicorn running on http://0.0.0.0:8000

# 4. Point your agent at http://localhost:8000/v1

Verify it's alive:

curl http://localhost:8000/status

Configuration

All options come from environment variables or a .env file (auto-loaded).

Point Your Agent At It

OpenCode

Add to opencode.json (or ~/.config/opencode/opencode.json):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "nvidia": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "UnlimitedNIM",
      "options": {
        "baseURL": "http://localhost:8000/v1",
        "apiKey": "anything-works"
      },
      "models": {
        "nvidia/nemotron-3.5-lightning-30b-a3b": {
          "name": "Nemotron 3.5 Lightning 30B"
        }
      }
    }
  }
}

Cline

1. Settings → API Provider → OpenAI Compatible
2. Base URL: http://localhost:8000/v1
3. API Key: anything (the proxy uses its own NVIDIA key)
4. Model: nvidia/nemotron-3.5-lightning-30b-a3b

Roo Code

1. Settings → API Configuration → Provider → OpenAI Compatible
2. Base URL: http://localhost:8000/v1
3. API Key: anything
4. Model: nvidia/nemotron-3.5-lightning-30b-a3b

Load Testing

load_test.py simulates heavy agentic traffic (100 concurrent requests, mixed priorities, two waves) and reports status counts, leaked 429s, and queue wait times:

python load_test.py                       # full run (40 + 60 waves)
python load_test.py --total 12 --wave1 5 --wave2 7   # quick smoke

For safe local runs without touching NVIDIA, start the included mock upstream and point the proxy at it:

python mock_upstream.py                    # terminal 1
NVIDIA_BASE_URL=http://localhost:9001/v1 python main.py   # terminal 2

Project Layout

app/
  config.py        # env-driven settings
  rate_limiter.py  # sliding-window limiter + priority queue + in-flight tracking
  proxy.py         # FastAPI app, streaming upstream, 429 retry, validation
main.py            # entry point
load_test.py       # agentic traffic simulator
mock_upstream.py   # fake NVIDIA upstream for local testing
tests/             # 43 tests covering all 34 test-case specs

Test Coverage

The suite implements the full 34-case spec — window resets, previous-minute in-flight tracking, priority ordering, starvation promotion, clock skew, upstream 429/500 handling, client disconnects, and the compliance layer (/v1/models, /status).

python -m pytest -q   # 43 passed