GitHub - davebcn87/pi-autoresearch: Autonomous experiment loop extension for pi

Try an idea, measure it, keep what works, discard what doesn't, repeat forever.

Inspired by karpathy/autoresearch. Works for any optimization target: test speed, bundle size, LLM training, build times, Lighthouse scores.

Quick start

pi install https://github.com/davebcn87/pi-autoresearch

What's included

Extension tools

/autoresearch command

Examples:

/autoresearch optimize unit test runtime, monitor correctness
/autoresearch model training, run 5 minutes of train.py and note the loss ratio as optimization target
/autoresearch off
/autoresearch clear

Keyboard shortcuts

UI

• Status widget — always visible above the editor: 🔬 autoresearch 12 runs 8 kept │ ★ total_µs: 15,200 (-12.3%) │ conf: 2.1×
• Confidence score — after 3+ runs, shows how the best improvement compares to the session noise floor. ≥2.0× (green) = likely real, 1.0–2.0× (yellow) = above noise but marginal, <1.0× (red) = within noise.
• Expanded dashboard — Ctrl+X expands the widget into a full results table with columns for commit, metric, status, and description.
• Fullscreen overlay — Ctrl+Shift+X opens a scrollable full-terminal dashboard. Shows a live spinner with elapsed time for running experiments.

Skill

autoresearch-create asks a few questions (or infers from context) about your goal, command, metric, and files in scope — then writes two files and starts the loop immediately:

Install

pi install https://github.com/davebcn87/pi-autoresearch

cp -r extensions/pi-autoresearch ~/.pi/agent/extensions/
cp -r skills/autoresearch-create ~/.pi/agent/skills/

Usage

1. Start autoresearch

/skill:autoresearch-create

The agent asks about your goal, command, metric, and files in scope — or infers them from context. It then creates a branch, writes autoresearch.md and autoresearch.sh, runs the baseline, and starts looping immediately.

2. The loop

The agent runs autonomously: edit → commit → run_experiment → log_experiment → keep or revert → repeat. It never stops unless interrupted.

Every result is appended to autoresearch.jsonl in your project — one line per run. This means:

• Survives restarts — the agent can resume a session by reading the file
• Survives context resets — autoresearch.md captures what's been tried so a fresh agent has full context
• Human readable — open it anytime to see the full history
• Branch-aware — each branch has its own session

3. Monitor progress

• Widget — always visible above the editor
• /autoresearch — full dashboard with results table and best run
• Escape — interrupt anytime and ask for a summary

Example domains

How it works

The extension is domain-agnostic infrastructure. The skill encodes domain knowledge. This separation means one extension serves unlimited domains.

┌──────────────────────┐     ┌──────────────────────────┐
│  Extension (global)  │     │  Skill (per-domain)       │
│                      │     │                           │
│  run_experiment      │◄────│  command: pnpm test       │
│  log_experiment      │     │  metric: seconds (lower)  │
│  widget + dashboard  │     │  scope: vitest configs    │
│                      │     │  ideas: pool, parallel…   │
└──────────────────────┘     └──────────────────────────┘

Two files keep the session alive across restarts and context resets:

autoresearch.jsonl   — append-only log of every run (metric, status, commit, description)
autoresearch.md      — living document: objective, what's been tried, dead ends, key wins

A fresh agent with no memory can read these two files and continue exactly where the previous session left off.

Configuration (optional)

Create autoresearch.config.json in your pi session directory to customize behavior:

{
  "workingDir": "/path/to/project",
  "maxIterations": 50
}

Confidence scoring

After 3+ experiments in a session, pi-autoresearch computes a confidence score — how the best improvement compares to the session's noise floor. This helps distinguish real gains from benchmark jitter, especially on noisy signals like ML training, Lighthouse scores, or flaky benchmarks.

How it works:

• Uses Median Absolute Deviation (MAD) of all metric values in the current segment as a robust noise estimator.
• Confidence = |best_improvement| / MAD. A score of 2.0× means the best improvement is twice the noise floor.
• Shown in the widget, expanded dashboard, and log_experiment output.
• Persisted to autoresearch.jsonl on each result for post-hoc analysis.
• Advisory only — never auto-discards. The agent is guided to re-run experiments when confidence is low, but the final keep/discard decision stays with the agent.

Backpressure checks (optional)

Create autoresearch.checks.sh to run correctness checks (tests, types, lint) after every passing benchmark. This ensures optimizations don't break things.

#!/bin/bash
set -euo pipefail
pnpm test --run
pnpm typecheck

How it works:

• If the file doesn't exist, everything behaves exactly as before — no changes to the loop.
• If it exists, it runs automatically after every benchmark that exits 0.
• Checks execution time does not affect the primary metric.
• If checks fail, the experiment is logged as checks_failed (same behavior as a crash — no commit, revert changes).
• The checks_failed status is shown separately in the dashboard so you can distinguish correctness failures from benchmark crashes.
• Checks have a separate timeout (default 300s, configurable via checks_timeout_seconds in run_experiment).

License

MIT