GitHub - varmabudharaju/chorus

Federated LoRA fine-tuning with mathematically exact aggregation — when residuals are folded into base weights every round (details).

Chorus is a framework for federated fine-tuning of large language models using LoRA adapters. Multiple clients train on their private data, submit adapter deltas to a central server, and receive back aggregated improvements — without sharing any raw data.

The key insight: standard FedAvg is broken for LoRA because avg(B @ A) != avg(B) @ avg(A). Chorus implements FedEx-LoRA (ACL/ICLR 2025), which provides exact federated aggregation by tracking and folding SVD residuals.

How It Works

Client 1 (private data)          Aggregation Server           Client 2 (private data)
┌─────────────────────┐       ┌─────────────────────┐       ┌─────────────────────┐
│  1. Train LoRA      │       │                     │       │  1. Train LoRA      │
│  2. Submit delta  ──┼──POST─┼→ Collect deltas     │←─POST─┼── 2. Submit delta   │
│                     │       │  FedEx-LoRA agg     │       │                     │
│                     │       │  Fold residuals     │       │                     │
│  3. Pull updated  ←─┼──GET──┼─ Serve result       │──GET──┼→ 3. Pull updated    │
│  4. Repeat          │       │  WS: round_complete │       │  4. Repeat          │
└─────────────────────┘       └─────────────────────┘       └─────────────────────┘

Honest tradeoffs

Chorus's claims hold under conditions that aren't always obvious from a quickstart. Before relying on any of them, read docs/honest-tradeoffs.md. Highlights:

• "Mathematically exact" aggregation holds when residuals are folded into base weights every round — the default server path, but not yet the eval harness (#19). More
• Differential privacy is per-submission Gaussian noise with a stateful accountant — but the accountant is opt-in: set --accountant-target-epsilon and --accountant-noise-multiplier on the server to bound the privacy budget. Without them, privacy loss accumulates unbounded. More
• "Byzantine defenses" are sanity checks against naive attackers (norm bound + z-score outlier). They will not stop an adaptive adversary. Real Byzantine-robust aggregation is on the Phase 2 roadmap. More
• API keys are global. No per-model scoping in v0.2.0. Run one server per trust boundary. More
• Heterogeneous-rank clients work on fedex-lora and crash on fedavg. More
• Alpha software: single-process server, in-memory rate limiter, filesystem storage, HTTP. Not hardened for multi-tenant production. More

Installation

With optional dependencies:

# For local LoRA training (PEFT + Transformers)
pip install "chorus-fl[peft]"

# For differential privacy
pip install "chorus-fl[privacy]"

# Everything
pip install "chorus-fl[all]"

From source:

git clone https://github.com/varmabudharaju/chorus.git
cd chorus
pip install -e ".[dev]"

Quick Start

1. Start the server

chorus server --model meta-llama/Llama-3.2-3B --min-deltas 3

2. Submit adapters from clients

from chorus import ChorusClient

client = ChorusClient(
    server="http://localhost:8080",
    model_id="meta-llama/Llama-3.2-3B",
)

# After your local LoRA training...
client.submit_delta(adapter_path="./my-adapter")

# Pull the aggregated global adapter
client.pull_latest(output_path="./updated-adapter")

client.close()

3. Run a simulation (no server needed)

# Compare FedAvg vs FedEx-LoRA
chorus simulate --clients 10 --rounds 5 --compare

Why FedEx-LoRA?

LoRA decomposes weight updates as W = B @ A (two low-rank matrices). When you naively average across clients:

avg(B_i @ A_i)  !=  avg(B_i) @ avg(A_i)

FedAvg produces mathematically inexact aggregation for LoRA. FedEx-LoRA fixes this:

1. Computes the exact weighted average of full-rank products B_i @ A_i
2. Uses SVD to get the optimal rank-r approximation (Eckart-Young theorem)
3. Tracks the residual between exact and approximate results
4. Folds residuals into base weights, making the combined result exact

CLI Reference

chorus server

Start the aggregation server.

chorus server --model <model-id> [options]

chorus submit

Submit a LoRA adapter delta to the server.

chorus submit --server <url> --adapter <path> [options]

chorus pull

Pull the latest aggregated adapter from the server.

chorus pull --server <url> --output <path> [options]

chorus train

Run the full federated training loop (train -> submit -> wait -> pull -> repeat).

chorus train --server <url> --model <hf-model-id> --dataset <dataset> [options]

chorus simulate

Run a simulated federation with synthetic data.

chorus simulate --clients 10 --rounds 5 --compare

chorus status

Show the current status of a Chorus server.

chorus status --server <url>

chorus export

Export a merged model (base + aggregated adapter) ready for deployment.

chorus export --server <url> --model <hf-model-id> --output ./merged/

Python SDK

ChorusClient

from chorus import ChorusClient

client = ChorusClient(
    server="http://localhost:8080",
    model_id="my-model",
    client_id="client-1",          # optional, auto-generated if omitted
    api_key="secret",              # optional, for authenticated servers
    dp_epsilon=1.0,                # optional, local differential privacy
    dp_delta=1e-5,                 # optional, DP delta parameter
    dp_max_norm=1.0,               # optional, DP clipping norm
    timeout=120.0,                 # optional, HTTP timeout in seconds
)

# Check server status
status = client.status()

# Submit a trained LoRA adapter
result = client.submit_delta(
    adapter_path="./my-adapter",   # PEFT adapter dir or .safetensors
    round_id=None,                 # None = current round
    dataset_size=5000,             # for weighted aggregation
)

# Submit raw tensors directly
result = client.submit_tensors(tensors={"layer.lora_A.weight": tensor_a, ...})

# Pull the latest aggregated adapter
client.pull_latest(output_path="./updated-adapter")

# Pull a specific round
client.pull_round(round_id=3, output_path="./round-3-adapter")

# Export merged model (requires chorus[peft])
client.export_model(
    base_model="meta-llama/Llama-3.2-3B",
    output_dir="./merged-model",
)

# Full training loop (requires chorus[peft])
client.train_loop(
    trainer=my_trainer,            # LoRATrainer instance
    rounds=5,
)

# Listen for round completion via WebSocket
for event in client.listen():
    print(f"Round {event['round_id']} complete!")

client.close()
# Or use as context manager:
# with ChorusClient(...) as client:
#     ...

API Endpoints

Architecture

chorus/
├── patterns.py              # Shared LoRA key patterns
├── exceptions.py            # Exception hierarchy (ChorusError, etc.)
├── server/
│   ├── app.py               # FastAPI endpoints + auth + async aggregation
│   ├── aggregation.py       # FedAvg + FedEx-LoRA (SVD) + Byzantine defenses
│   ├── storage.py           # Filesystem storage for deltas, base weights, round state
│   ├── weight_manager.py    # Residual folding into base weights
│   ├── ws.py                # WebSocket connection manager
│   └── privacy.py           # Gaussian DP mechanism + L2 clipping
├── client/
│   ├── sdk.py               # ChorusClient (submit, pull, listen, train_loop, export)
│   ├── trainer.py           # LoRATrainer wrapper for HF PEFT
│   └── delta.py             # LoRA matrix extraction from PEFT adapters
├── cli/
│   └── main.py              # Click CLI with error handling
└── simulate/
    └── runner.py            # Synthetic multi-client federation runner

Security Features

Chorus includes several security mechanisms for production deployments:

• Authentication — Bearer token auth via --api-key (supports multiple keys)
• Differential privacy — Per-submission Gaussian DP with global L2 clipping (--dp-epsilon), plus a stateful PrivacyAccountant (RDP composition via Google's dp-accounting, opacus fallback). Set --accountant-target-epsilon and --accountant-noise-multiplier on the server to halt before you exceed your budget; without those, privacy loss accumulates unbounded across rounds. The eval harness does not yet apply DP (#20); the server path does. Details.
• Sanity-check defenses against naive attackers — L2 norm bounding (--norm-bound) and z-score outlier detection (--outlier-threshold) reject deltas with absurd magnitude or those several standard deviations from the round's median. These catch random-noise injection and trivial corruption; they do not stop coordinated attackers staying under the bound, label-flipping at the task level, or gradient-inversion attacks (details). Real Byzantine-robust aggregation (Krum, coordinate-wise median) is on the Phase 2 roadmap.
• Rate limiting — Per-IP request throttling via --rate-limit
• safetensors only — All weight serialization uses safetensors format (no pickle deserialization)

Note: Chorus serves over HTTP. For production, deploy behind a TLS-terminating reverse proxy (nginx, Caddy, etc.).

Aggregation Strategies

Configuration Examples

Secure production server

chorus server \
  --model meta-llama/Llama-3.2-3B \
  --min-deltas 5 \
  --api-key $SECRET_KEY_1 \
  --api-key $SECRET_KEY_2 \
  --dp-epsilon 2.0 \
  --accountant-target-epsilon 8.0 \
  --accountant-noise-multiplier 1.1 \
  --norm-bound 10.0 \
  --outlier-threshold 3.0 \
  --rate-limit 60 \
  --base-weights ./base-model.safetensors

Note: --dp-epsilon alone applies noise per submission but does NOT bound cumulative privacy loss. Pair it with --accountant-target-epsilon and --accountant-noise-multiplier (the example above) so the server refuses further submissions from a client once the budget is exhausted. See docs/honest-tradeoffs.md#differential-privacy.

Client with local DP

client = ChorusClient(
    server="http://chorus.internal:8080",
    model_id="meta-llama/Llama-3.2-3B",
    api_key="my-secret-key",
    dp_epsilon=1.0,        # strong local DP
    dp_max_norm=1.0,       # clip before noising
)

Full training loop

chorus train \
  --server http://localhost:8080 \
  --model meta-llama/Llama-3.2-3B \
  --dataset wikitext \
  --rounds 10 \
  --lora-rank 16

Production readiness

Chorus is alpha software in v0.2.0. Suitable for research, small internal federations, and benchmarks; not hardened for multi-tenant production. Single-process FastAPI server, in-memory rate limiter, filesystem storage, HTTP (terminate TLS at a reverse proxy). API keys are global — run one server per trust boundary. See docs/honest-tradeoffs.md#production-readiness for the full list of caveats and what's on the roadmap to fix each.

Examples

See the examples/ directory:

• quickstart.py — Basic 2-client workflow with synthetic adapters
• health_metrics/federated_health.py — Multi-hospital federated training simulation with DP

Development

git clone https://github.com/varmabudharaju/chorus.git
cd chorus
pip install -e ".[dev]"

# Run tests (165 tests)
pytest tests/ -v

# Run benchmarks
python benchmarks/benchmark.py

Contributing

Contributions are welcome! Here's how to get started:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/my-feature)
3. Make your changes and add tests
4. Run the test suite (pytest tests/ -v)
5. Submit a pull request

Please open an issue first to discuss significant changes.

License

Apache 2.0 — see LICENSE for the full text.