GitHub - ColonelToad/Rally

Rally: Real-Time Dual-Arm Robotic Table Tennis System

Rally is a high-performance robotic table tennis simulation combining deterministic C++ real-time control loops (1kHz physics, 500Hz per-arm) with Python-based LLM orchestration. The system demonstrates ownership arbitration without learning, EKF-based ball prediction (validated against real robot data), and adaptive arm strategy via recursive least squares.

Quick Links

• Architecture & Design: See docs/TECHNICAL_WRITEUP.md (3.5k words)
• Design Philosophy: docs/PHILOSOPHY.md — read first for context
• Testing & Validation: docs/TESTING.md
• Embedded Deployment: docs/EMBEDDED_NOTES.md

System Architecture

Rally splits execution across C++ (real-time) + Python (orchestration), communicating entirely via ZeroMQ to ensure process isolation and determinism.

C++ Real-Time Engine (1kHz + 500Hz)

Key Algorithms:

• Ownership Arbiter: Hysteresis-based spatial partitioning (no learning)
• EKF Ball Predictor: Drag model with 4.1mm validation error on real robot data
• AnalyticalIK: 7-DOF Franka Panda with joint limits
• RLS Adaptive Strategy: Per-arm bias learning (θ = [target_offset_y, aggression_factor, reaction_margin])

Python Orchestrator (Viewer + LLM)

• MuJoCo Passive Viewer: Renders the sim, drives the human paddle over ZMQ, runs the 3-2-1 warm-up countdown, live score HUD, and the watchdogs (frame health, blind-bridge detection). Closing the window freezes the sim, prints the coach's post-game review in the terminal, and stops every engine process
• LLM Brain (optional): Qwen2.5 via llama.cpp — per-rally tactical calls (StrategyCommand) and the post-game review; all inference is serialized behind a lock
• Streamlit Dashboard (future): Telemetry visualization

Technical Highlights

No Dynamic Allocation in Hot Paths

// All control-path data structures are fixed-size
class EkfBallPredictor {
    Eigen::Vector6d state_;
    Eigen::Matrix6d P_;
};

Unified Binary Logging (One Schema, One Replayer)

• 64-byte fixed-size LogRecord written by every subsystem
• Lock-free per-thread SPSC buffers
• 5ms periodic flush to disk
• Supports live dashboards + offline analysis

Real-World Validation

• DeepMind Dataset: EKF tested on 200 real ball trajectories → 4.1mm mean error
• Fixed-Point Path: Q32.32 math validated for embedded FPU-less targets
• Integration Tests: All 4 processes run concurrently, telemetry logged to disk

ZeroMQ Endpoints

Demo

Watch the demo — dual-arm rallies with ownership arbitration, human paddle play (movement, timed swing, scoring), the 3-2-1 warm-up countdown, live score HUD, and the coach's post-game review in the terminal.

Note on lag: the visible frame stutter comes from running the renderer through WSLg's software rasterizer (llvmpipe) on this machine — a display-stack constraint, not the control system. The physics and control loops run deterministically at 1kHz/500Hz in C++ regardless of frame rate, and a native (non-WSL) build renders this scene smoothly.

Getting Started

Prerequisites

# Ubuntu/Debian
sudo apt-get install cmake g++ libeigen3-dev libzmq3-dev

# Python
pip install mujoco llama-cpp-python

Build & Run

mkdir build && cd build
cmake ..
make

# Terminal 1: Physics loop (1kHz)
./mujoco_bridge

# Terminal 2: Ownership arbiter + RLS (100Hz)
./coordination_bus

# Terminal 3 & 4: Per-arm control (500Hz each, with core affinity)
./arm_controller --side left --core 2
./arm_controller --side right --core 3

# View logs
ls logs/rally_telemetry*.log   # Binary telemetry from each process

# Terminal 5: Human player viewer (60 FPS + LLM coach)
python viewer.py

Human Player Controls

The viewer drives a kinematic racket (human_paddle mocap body) over ZMQ. Position it to intercept the incoming ball, then time your swing:

The viewer launches with MuJoCo's UI panels hidden (press Tab to show them) and overlays a live HUD: session score, rally state, the LLM coach's current tactical call, and each arm's live RLS parameters.

Match Countdown & Scene

On launch the viewer freezes the simulation and runs a 3-2-1 warm-up countdown (shader compile + first slow software-rendered frames happen before the match starts), then resumes into an opening serve. RALLY_COUNTDOWN=0 disables it. The robot arms stand on pedestals behind the table edge, mounted at table height.

Post-Game Review (terminal)

Closing the viewer window ends the session: the simulation freezes, the coach writes a review of your match (score, rallies, swing count, and how each arm's RLS parameters adapted), and it prints to the viewer terminal after the shutdown message. The on-window HUD stays minimal — countdown and live score only — since MuJoCo's text overlays are small and stack.

Graphics / Performance Notes (WSL)

• Default rendering is Mesa's llvmpipe (CPU): stable everywhere. The viewer auto-scales its raster threads (LP_NUM_THREADS) and defaults to 30fps; physics always runs at 1kHz in C++ regardless.
• WSLg's D3D12 GPU passthrough (GALLIUM_DRIVER=d3d12) is faster but has frozen entire systems on Intel iGPUs during sustained rendering. It is therefore opt-in: RALLY_D3D12=1 mujoco_env/bin/python viewer.py — save your work first. Override either path's fps with RALLY_RENDER_FPS=45.
• If the viewer ever dies without cleanup (hard crash), run ./stop_rally.sh to reap orphaned engine processes.

Build Options

# Fixed-point math for embedded targets (no FPU)
cmake .. -DRALLY_FIXED_POINT=ON

# Include LLM orchestrator + Streamlit (adds Python dependency)
cmake .. -DRALLY_ONLINE_FEATURES=ON

Testing

# Unit tests
ctest --output-on-failure

# Integration test (all 4 processes + telemetry validation)
bash tests/integration/test_rls_pipeline.sh