A history of Harness Hacks

3 min read Original article ↗

← → space · n notes · b blank · p presenter · f fullscreen · g go · t reset clock · ? help

A history of
Harness Hacks

how will we prompt AGI?

tools as code thousands of integrations one continuous agent same work on every surface multiplayer, async steer mid-task wherever you work Slack, Teams, APIs, MCP

Between the model
and the world: harness.

harness model world context → action ← result / observation

Language Models are
Few-Shot Learners.

sea → morze dog → pies cat → weights fixed GPT-3 kot

Prompt Programming:
examples locate a task.

sea → morze dog → pies Translate to Polish: model summarize classify extract rhyme count translate both locate a task the model already knows

AI Chains:
one prompt per step.

result result prompt 1 prompt 2 prompt 3 each result is read, fixed and passed on

Large Language Models are
Zero-Shot Reasoners.

GSM8K maths · text-davinci-002 accuracy answer only 10.4% “Let's think step by step” working → answer 40.7%

ReAct:
reason, act, observe.

reason act observe model proposes world replies

Tree of Thoughts:
try another branch.

problem try A try B try C continue

Grammar-based
sampling.

want Alice as JSON Sure!… parser rejects constrain each token name: string {"name":"Bob"} parses; wrong name illustrative

Large Language Models
as Optimizers.

OPRO · PaLM 2-L answers GSM8K maths candidate prompt training score model revises Held-out test accuracy · baseline → best reported 71.8% → 80.2%

Large Language
Monkeys.

DeepSeek-Coder-V2-Instruct · SWE-bench Lite 1 → 250 tries selection external check ≥1 passing patch: 15.9% → 56% Coverage ≠ selected success

Multi-agent research:
fresh windows.

lead splits the task each worker: a fresh window, brief and tools lead worker worker worker Anthropic: multi-agent ≈ 15× chat tokens (typical)

Live-SWE-agent:
rewrite the harness.

File tool returns too much code · illustration model writes fixed weights editable external code reads 100 lines 201–300 → next 301 check protected

Speculative Programmatic
Tool Calling.

usual generate codestart calls overlap generate codereuse results start calls eligible calls only

Three trajectories.

shape of work · one prompt → a graph chain loop tree graph one prompt the model eats the harness · prompt tricks → the world reasoning format prompting harness execution climb the world scale · one call → hundreds of agents one prompt tries subagents swarm

Turn most tasks into
a hill climb.

objective experiments score keep prompt · tools · memory · routing the gradient is the scarce artifact

Work becomes
a graph.

brief attempt A attempt B attempt C check best fail → retry branch

The world retreats
into the model.

world tools · context harness model stays outside permission the check intent

A civilization
of experts.

prompt model tries subagents graph city civilization

How will we
prompt AGI?

  • Probably through an agent that makes the case.
  • The prompt might be short.
  • The work behind it could be civilization-sized.