GitHub - byt3bl33d3r/figaro: Orchestrate fleets of Claude Code & Claude Computer Use agents across containers, VMs, and physical devices. Live desktop streaming, intelligent task delegation, and multi-channel human-in-the-loop gateway. Schedule tasks with self-healing and self-learning.

Figaro orchestrates fleets of Claude computer use agents that automate workflows on full desktop environments. Agents run inside containerized Linux desktops or connect to any VNC-accessible machine -- remote servers, cloud VMs, or physical workstations. All desktops are live-streamed to a central dashboard, and a supervisor agent handles task delegation. Everything can be managed conversationally through external channels like Telegram.

Figaro can connect to any accessible desktop -- local machines, remote servers, cloud VMs, or physical workstations running macOS, Windows, or Linux. Desktops are added from the UI with a connection URL (vnc://, rdp://, ssh://, telnet://, ws://, or wss://), and the supervisor agent can observe and interact with any connected desktop via screenshots, typing, clicking, and key presses. Desktop streaming uses Apache Guacamole (guacd + guacamole-common-js) for protocol-agnostic remote access.

The system is built for long-running tasks that take minutes to hours. All services communicate over NATS (pub/sub + JetStream for durable task events). A supervisor agent handles task optimization and delegation.

You can also manage everything by chatting with the supervisor agent through the gateway -- for example, via Telegram. Send it natural language instructions to create tasks, schedule recurring jobs, check worker status, or ask questions about running tasks. The supervisor understands the full system and can delegate work to workers, inspect desktops via VNC, and report back results, all through a conversational interface.

Table of Contents

• Quick Start
• Advanced Setup
• Connecting External Desktops
• Scheduled Tasks
• Self-Healing
• Self-Learning
• Agent Memory
• Gateway
• Security
• Architecture
• Services
• Development Setup
• Configuration
• NATS Subject Design
• Message Flows
• Testing
• Contributing
• Credits

UI Action shot

Quick Start

curl -fsSL https://raw.githubusercontent.com/byt3bl33d3r/figaro/main/install.sh | bash

Or clone and run directly:

git clone https://github.com/byt3bl33d3r/figaro.git && cd figaro && ./install.sh

By default the install script uses the prod-local overlay, Figaro will be available at http://localhost:8000.

Prerequisites

• Docker and Docker Compose -- on Linux the install script will install Docker automatically via get.docker.com. On macOS you must install Docker Desktop manually before running the script.
• Claude credentials (~/.claude.json and ~/.claude/.credentials.json) -- created by running claude and signing in. On macOS (assuming you have a Claude Code subscription) once logged in, you can export the credentials file for use in containers with the following command:

  security find-generic-password -s "Claude Code-credentials" -w > ~/.claude/.credentials.json

Optional

• An OpenAI API key (used for patchright-cli transcription functionality)
• A Telegram bot token (enables chatting with the supervisor agent via Telegram for task submission, status checks, and notifications). Create a bot via @BotFather and set GATEWAY_TELEGRAM_BOT_TOKEN and GATEWAY_TELEGRAM_ALLOWED_CHAT_IDS in your .env

Advanced Setup

cp .env.example .env

# Generate an encryption key for VNC password storage (required)
echo "FIGARO_ENCRYPTION_KEY=$(openssl rand -hex 16)" >> .env

# Optional: set an OPENAPI_API_KEY
# OPENAI_API_KEY=sk-proj-example
# Optional: set Telegram gateway variables for notifications and task submission
# GATEWAY_TELEGRAM_BOT_TOKEN=your-bot-token
# GATEWAY_TELEGRAM_ALLOWED_CHAT_IDS=["your-chat-id"]

VNC passwords are encrypted at rest in PostgreSQL using pgcrypto. The FIGARO_ENCRYPTION_KEY is used for symmetric encryption and must be set before starting the orchestrator. The install.sh script generates this automatically.

The base docker/docker-compose.yml defines all shared services but does not expose ports. Choose an overlay for your deployment scenario:

# Production, localhost-only (recommended) -- ports bound to 127.0.0.1
docker compose -f docker/docker-compose.yml -f docker/docker-compose.prod-local.yml up --build

# Development -- localhost ports + a desktop service for testing VNC
docker compose -f docker/docker-compose.yml -f docker/docker-compose.dev.yml up --build

docker compose -f docker/docker-compose.yml -f docker/docker-compose.prod.yml up --build

Open http://localhost:8000.

This starts PostgreSQL, NATS (port 8443), guacd (Guacamole daemon), the orchestrator (port 8000), 2 workers, 2 supervisors, and the gateway. The supervisor service uses the same figaro-worker binary started with the --supervisor flag.

Scaling

docker compose -f docker/docker-compose.yml -f docker/docker-compose.prod-local.yml up --build --scale worker=4 --scale supervisor=3

Teardown

docker compose -f docker/docker-compose.yml -f docker/docker-compose.prod-local.yml down       # Stop services
docker compose -f docker/docker-compose.yml -f docker/docker-compose.prod-local.yml down -v    # Stop and remove all data

Connecting External Desktops

Figaro is not limited to its own containerized workers. Any machine with a VNC server can be connected as a desktop.

From the UI

Click "Add Desktop" in the dashboard header. Provide a worker ID, a desktop URL, and select the OS type. Supported URL schemes:

• vnc://user:password@hostname:5901 -- direct TCP VNC connection
• rdp://user:password@hostname:3389 -- RDP connection
• ssh://user:password@hostname:22 -- SSH connection
• telnet://user:password@hostname:23 -- telnet connection
• ws://hostname:6080 -- WebSocket (legacy noVNC-compatible)
• wss://hostname:6080 -- WebSocket over TLS (legacy noVNC-compatible)

Credentials can be embedded in the URL or entered separately. Connected desktops appear in the live desktop grid and can be viewed, screenshotted, and interacted with by the supervisor agent's VNC tools. SSH and telnet connections are accessible via the supervisor's terminal tools (ssh_run_command, telnet_run_command).

From Environment Variables

Pre-configure desktops at startup via the FIGARO_DESKTOP_WORKERS environment variable:

FIGARO_DESKTOP_WORKERS='[{"id": "mac-studio", "novnc_url": "vnc://user:pass@192.168.1.50:5900", "metadata": {"os": "macos"}}]'

Agent Upgrade Path

Desktop-only entries act as placeholders. When a worker agent connects with a matching ID, the desktop is automatically upgraded to a full agent worker capable of receiving tasks. When the agent disconnects, the desktop reverts to view-only mode rather than disappearing. This means you can pre-register your desktops and have agents attach and detach dynamically.

This is useful when you want to connect existing physical machines or VMs to Figaro without running the full containerized worker stack. For example, you can point Figaro at your Mac Mini's VNC server, and the supervisor can already observe and interact with its screen. Later, run the figaro-worker agent on that machine to give it full task execution capabilities -- the desktop entry upgrades seamlessly without any reconfiguration. If you install patchright-cli on that machine it becomes particularly handy: it's a standalone browser automation CLI that speaks the same claude-agent-sdk protocol, so you can drop it onto any machine with a browser and instantly level up your browser automation tasks.

Scheduled Tasks

Figaro supports cron-like scheduling for recurring tasks. Scheduled tasks are managed through the UI or directly by chatting to Figaro via Telegram or whatever channel is configured in the Gateway.

Each scheduled task has:

• Prompt -- the task instruction sent to the agent
• Cron expression -- standard cron syntax for scheduling (e.g., 0 9 * * * for daily at 9 AM)
• Start URL -- optional URL the worker's browser should navigate to before starting
• Enabled/disabled toggle -- pause and resume without deleting
• Manual trigger -- execute immediately regardless of schedule or enabled state
• Notify on completion via Gateway -- send a notification to configured gateway channels (e.g., Telegram) when the task completes or fails
• Self-learning -- optional automatic prompt optimization after each run

Scheduled tasks are assigned to supervisors, which delegate to workers. The scheduler checks for due tasks every 60 seconds and assigns them to idle supervisors (or queues them if none are available).

NATS API

figaro.api.scheduled-tasks                # List all scheduled tasks
figaro.api.scheduled-tasks.get            # Get a specific scheduled task
figaro.api.scheduled-tasks.create         # Create a new scheduled task
figaro.api.scheduled-tasks.update         # Update an existing scheduled task
figaro.api.scheduled-tasks.delete         # Delete a scheduled task
figaro.api.scheduled-tasks.toggle         # Enable or disable a scheduled task
figaro.api.scheduled-tasks.trigger        # Trigger immediate execution

When a task fails, the orchestrator can automatically create a "healer" task that analyzes the failure and retries with an improved approach. This is designed for browser automation where failures are often transient -- an element didn't load in time, a page layout changed, or the agent navigated to the wrong place.

How It Works

1. A worker publishes an error to figaro.task.{id}.error
2. The orchestrator evaluates whether to heal based on a priority chain:
   • Task-level: the self_healing flag on the specific failed task
   • Scheduled task-level: the self_healing field on the associated scheduled task (if any)
   • System-level: FIGARO_SELF_HEALING_ENABLED environment variable (default: true)
3. If healing is enabled and retries remain (< FIGARO_SELF_HEALING_MAX_RETRIES, default: 2), a healer task is created
4. The healer task contains the original prompt, the error message, and the conversation history (last 50 messages)
5. A supervisor analyzes the failure and decides:
   • Recoverable (element not found, timing issues, navigation errors): delegates to a worker with an improved prompt. May use VNC tools to inspect the desktop state first.
   • Unrecoverable (invalid credentials, service down, fundamental approach failure): explains why and does not retry.
6. If the retried task also fails and retries remain, another healer is created (up to max_retries)

Loop Prevention

• Tasks with source="healer" or source="optimizer" are never healed -- only original tasks trigger healing
• The retry chain is tracked via original_task_id and retry_number to enforce the max retries limit

Configuration

Self-Learning

Scheduled tasks with self_learning enabled automatically optimize their prompts after each completed run. The system analyzes how the task actually executed and rewrites the prompt to be more effective next time.

How It Works

1. A worker completes a scheduled task
2. The orchestrator checks: does the task have a scheduled_task_id, source="scheduler", and self_learning=True?
3. If so, it retrieves the conversation history, filters to key message types (assistant, tool_result, result), and caps at the last 50 messages
4. An optimization task is created with the current prompt and conversation history, then assigned to an idle supervisor
5. The supervisor analyzes what happened during execution and calls update_scheduled_task to save an improved prompt
6. The next scheduled run uses the improved prompt

Design Details

• Optimization runs fire-and-forget as a background task -- failures are logged but don't affect the main task completion flow
• Optimization runs after every completed scheduled task run (no throttling)
• The supervisor is instructed to only update the prompt field, preserving schedule, enabled state, and start URL settings
• Self-learning is toggled per scheduled task via a checkbox in the UI

Agent Memory

Figaro has a persistent memory system that lets agents store and retrieve learned knowledge across task executions. Memories are stored in PostgreSQL with hybrid search (BM25 full-text via ParadeDB + pgvector semantic similarity), accessed via NATS request/reply through the agents' Python sandbox.

How It Works

Both supervisors and workers have access to memory functions (figaro.save_memory(), figaro.search_memories(), figaro.list_memories(), figaro.delete_memory()) available in their python_exec tool via a built-in figaro module. Claude Code's built-in Memory tool is disabled in favor of this custom implementation.

The supervisor workflow integrates memory at two points:

• Before delegation: searches memories for relevant context (site tips, user preferences, past errors) and injects it into the worker's prompt
• After task completion: saves any new learnings as memories for future tasks

Memory Structure

Each memory has:

Standard collections:

• sites -- site-specific knowledge (URLs, login flows, navigation patterns, cookie banners)
• users -- user preferences and requirements
• errors -- failure patterns and known fixes
• workflows -- multi-step procedures for recurring tasks

Search

Memory search uses Reciprocal Rank Fusion (RRF) combining BM25 full-text search and pgvector cosine similarity. BM25 receives 2x weight in the fusion. If no OpenAI API key is configured, embeddings are skipped and search falls back to BM25 only.

Memories are deduplicated by (collection, content_hash) -- saving the same content to the same collection updates the existing record.

NATS API

figaro.api.memories.save                  # Save a memory (upsert by content hash)
figaro.api.memories.search                # Hybrid BM25 + vector search
figaro.api.memories.list                  # List memories (optional collection filter)
figaro.api.memories.delete                # Delete a memory by ID

Configuration

Gateway

The gateway is a channel-agnostic messaging service that routes messages between external communication channels (Telegram, future: WhatsApp, Slack, etc.) and the orchestration system. Users can send natural language instructions to create tasks, schedule jobs, check status, or ask questions -- all through a conversational interface.

Messages can include attachments (images, documents, etc.) up to 20 MB, which are forwarded to the supervisor agent alongside the text prompt.

Voice Message Transcription

The gateway supports voice message transcription. When a user sends a voice or audio message, it is automatically transcribed to text and processed as a regular task -- no special handling needed from the user's perspective.

1. User sends a voice/audio message via Telegram
2. The gateway downloads the audio file from Telegram
3. Audio is converted to raw PCM (16-bit, 16kHz, mono) via ffmpeg
4. PCM data is streamed in 100ms chunks over a WebSocket to Claude's STT endpoint (/api/ws/speech_to_text/voice_stream)
5. The endpoint returns transcript segments which are assembled into the final text
6. The transcribed text follows the exact same path as a typed message -- published to NATS for task creation and supervisor delegation

Authentication uses a Claude CLI OAuth token read from the credentials file (~/.claude/.credentials.json). The token is loaded fresh on every request, so credential rotation is handled automatically. ffmpeg must be available in the gateway container's PATH (included in the Docker image).

Configuration

Security

Figaro is designed for trusted, isolated environments -- private Docker networks, and in "production" over encrypted overlay networks like Tailscale, Headscale, or Nebula. See SECURITY.md for a full catalogue of known security trade-offs, attack vectors, and the reasoning behind them.

Architecture

                     +--------------------+
                     |    NATS Server     |
                     |  (+ JetStream)     |
                     |  (+ WebSocket)     |
                     +---------+----------+
       +----------+-----------+-----------+--------------+
       |          |           |           |              |
  +----+----+ +---+----+ +---+-----+ +---+----------+ +-+----------+
  | Worker  | |Orchestr| |Supervis | |   Gateway    | |    UI      |
  |  (x N)  | |  ator  | |   or    | |  (channels)  | |   (SPA)    |
  +---------+ +---+----+ +---------+ +--------------+ +------------+
                  |
             +----+---------------+
             |       guacd        |
             | (Apache Guacamole) |
             +--------------------+

All services communicate via NATS (pub/sub + JetStream for durable task events). The UI connects to NATS via WebSocket (nats.ws) for both real-time events and mutations (request/reply). HTTP endpoints are minimal: GET /api/config (NATS URL discovery), GET /api/guacamole/token (encrypted connection tokens), and WS /guacamole/webSocket (Guacamole WebSocket tunnel via guapy).

Services

Orchestrator (figaro/)

FastAPI application that manages task lifecycle, worker registry, and scheduling. Serves the built UI as static files. Handles all NATS API request/reply operations (task CRUD, scheduled tasks, help requests, VNC/SSH/telnet proxy). Mounts a guapy Guacamole WebSocket server for desktop streaming. Persists state to PostgreSQL with Alembic migrations.

Stack: Python 3.14, FastAPI, SQLAlchemy (async), asyncpg, asyncvnc, guapy, asyncssh, telnetlib3, Pillow

Worker (figaro-worker/)

A Claude computer use agent that executes browser automation tasks via the claude-agent-sdk. The agent sees and interacts with a desktop like a human would -- taking screenshots, moving the mouse, clicking elements, typing text, and navigating between applications. It has its own set of skills (custom tools) and can run shell commands. Task progress is streamed to JetStream.

The worker is a standalone service that can run on any machine with a desktop environment. In Docker, it runs inside a containerized Linux desktop (Fluxbox + TigerVNC + Chromium + noVNC) provided by the container image, but it can also run directly on a physical or virtual machine -- see Connecting External Desktops.

The same binary also runs as a supervisor when started with the --supervisor flag. In supervisor mode, the agent receives complex tasks and delegates them to workers. It can directly observe and interact with any connected desktop via VNC tools -- taking screenshots, typing, clicking, and pressing keys -- without delegating a full task. It can also execute commands on workers with SSH or telnet connections. Uses SDK-native custom tools backed by NATS request/reply for delegation and task management. Supports blocking delegation with inactivity-based timeouts.

Stack: Bun (compiled native binary), @anthropic-ai/claude-agent-sdk, NATS

Legacy Python implementations of both the worker and supervisor exist in legacy/ for reference.

Gateway (figaro-gateway/)

Channel-agnostic messaging gateway that routes messages between external communication channels and the orchestration system. Currently supports Telegram. Implements a Channel protocol for adding new channels (WhatsApp, Slack, etc.) with minimal boilerplate.

Stack: Python 3.12+, python-telegram-bot, NATS

UI (figaro-ui/)

React single-page application providing a dashboard with live desktop grid (Guacamole viewers), event stream, chat input for task submission, scheduled task management, and help request handling. Connects directly to NATS via WebSocket for all communication. Desktop streaming uses guacamole-common-js with auto-reconnect and display scaling.

Stack: React 18, TypeScript, Vite, Zustand, Tailwind CSS, guacamole-common-js, nats.ws

Shared NATS Library (figaro-nats/)

Reusable Python package providing a typed NATS client wrapper (NatsConnection) with JSON serialization, auto-reconnect, and methods for Core NATS and JetStream operations. Also provides subject constants and stream configuration.

Stack: Python 3.12+, nats-py, Pydantic

Patchright CLI (patchright-cli/)

Browser automation CLI tool built on Patchright (a Playwright fork for undetected browser automation). Uses a daemon-per-session architecture with Unix socket communication. Installed inside worker containers to provide browser automation capabilities.

Stack: Python 3.14, Patchright, OpenAI SDK (for audio transcription)

Development Setup

Dev Container (Recommended)

The repository includes a VS Code Dev Container configuration with all dependencies pre-installed:

• Python 3.14, Node.js, Bun, uv
• Desktop environment (VNC) for local testing
• Docker-outside-of-Docker for container builds
• Claude Code CLI

Open the repository in VS Code and select "Reopen in Container."

Manual Setup

Each service uses uv for Python dependency management:

# Shared NATS library (install first -- other services depend on it)
cd figaro-nats && uv sync --frozen

# Orchestrator
cd figaro && uv sync --frozen

# Worker + Supervisor (Bun)
cd figaro-worker && bun install

# Gateway
cd figaro-gateway && uv sync --frozen

# UI
cd figaro-ui && npm install

Running Services Individually

# Orchestrator (port 8000)
cd figaro && uv run figaro

# Worker
cd figaro-worker && bun run dev

# Supervisor (same binary, --supervisor flag)
cd figaro-worker && bun run dev -- --supervisor

# Gateway
cd figaro-gateway && uv run figaro-gateway

# UI dev server (port 3000)
cd figaro-ui && npm run dev

Database Migrations

cd figaro
uv run alembic upgrade head                              # Apply migrations
uv run alembic revision --autogenerate -m "description"  # Create new migration

Configuration

Environment Variables

NATS Configuration

The NATS server runs with JetStream enabled, WebSocket on port 8443 (no TLS), and HTTP monitoring on port 8222. See docker/nats.conf for the full configuration.

NATS Subject Design

Registration and Presence (Core NATS)

figaro.register.worker                    # Worker registration
figaro.register.supervisor                # Supervisor registration
figaro.register.gateway                   # Gateway registration
figaro.deregister.{type}.{id}             # Graceful disconnect
figaro.heartbeat.{type}.{id}              # Periodic liveness

Task Assignment (Core NATS, point-to-point)

figaro.worker.{worker_id}.task            # Assign task to specific worker
figaro.supervisor.{supervisor_id}.task    # Assign task to specific supervisor

Task Events (JetStream, TASKS stream)

figaro.task.{task_id}.assigned            # Task assigned
figaro.task.{task_id}.message             # Streaming SDK output
figaro.task.{task_id}.complete            # Task completed
figaro.task.{task_id}.error               # Task failed

Help Requests (Core NATS)

figaro.help.request                       # New help request
figaro.help.{request_id}.response         # Response to help request

Broadcasts (Core NATS)

figaro.broadcast.workers                  # Updated workers list
figaro.broadcast.supervisors              # Updated supervisors list
figaro.broadcast.task_healing             # Task healing event

API (NATS request/reply)

figaro.api.delegate                       # Delegate task to worker
figaro.api.workers                        # List workers
figaro.api.tasks                          # List tasks
figaro.api.tasks.get                      # Get task
figaro.api.tasks.create                   # Create task
figaro.api.tasks.search                   # Search tasks
figaro.api.supervisor.status              # Supervisor status
figaro.api.scheduled-tasks                # List scheduled tasks
figaro.api.scheduled-tasks.{get,create,update,delete,toggle,trigger}
figaro.api.help-requests.respond          # Respond to help request
figaro.api.help-requests.dismiss          # Dismiss help request
figaro.api.vnc                            # VNC operations (screenshot, type, key, click)
figaro.api.ssh                            # SSH operations (run_command)
figaro.api.telnet                         # Telnet operations (run_command)
figaro.api.memories.save                  # Save a memory
figaro.api.memories.search                # Search memories (hybrid BM25 + vector)
figaro.api.memories.list                  # List memories
figaro.api.memories.delete                # Delete a memory

Gateway (Core NATS)

figaro.gateway.{channel}.send             # Send message via channel
figaro.gateway.{channel}.task             # Task from channel
figaro.gateway.{channel}.question         # Ask question via channel
figaro.gateway.{channel}.register         # Channel gateway registers availability

Message Flows

Task Submission (UI to Worker)

1. UI sends NATS request to figaro.api.tasks.create
2. Orchestrator creates the task, claims an idle worker, publishes to figaro.worker.{id}.task
3. Worker executes the task with the Claude Agent SDK
4. Worker streams messages via JetStream (figaro.task.{id}.message)
5. Worker publishes completion via JetStream (figaro.task.{id}.complete)
6. Orchestrator updates the database and sets the worker idle

Help Request Flow (Human-in-the-Loop)

1. Worker or supervisor publishes to figaro.help.request
2. Orchestrator creates the help request and broadcasts to the UI
3. Gateway routes the help request to a channel (e.g., Telegram)
4. First responder (UI or channel) replies
5. Response is routed back to the requesting agent

Testing

Python Services

cd figaro && uv run pytest              # Orchestrator
cd figaro-nats && uv run pytest         # Shared library
cd figaro-gateway && uv run pytest      # Gateway

Linting and formatting:

uv run ruff check .
uv run ruff format .

UI

cd figaro-ui
npm run test           # Run tests
npm run test:watch     # Watch mode
npm run build          # TypeScript check + production build

Worker (Bun)

cd figaro-worker && bun test

Contributing

Pull requests are welcome! Issues are disabled on this repository -- if you've found a bug or have a feature request, please start a thread in Discussions first. Once the approach is agreed upon, feel free to open a PR.

Credits

• OpenClaw for design inspiration around some agent control primitives (e.g. loop control, cost tracking etc..)
• Apache Guacamole & Guapy
• QMD for the agent memory RAG implementation