GitHub - hexcreator/computer-use-runtime: Production-oriented Rust computer-use runtime

Production-oriented Rust runtime for computer-use agents: pluggable backends, strict action schemas, policy and human approval gates, and reproducible conformance artifacts.

Target repository: xayhemLLC/computer-use-runtime.

Why This Project

Computer-use tooling is often demo-only and backend-specific. This project focuses on stable interfaces and operational guardrails:

one action contract across native, mock, and Playwright backends
deterministic simulation path for CI and local tests
auditable conformance reports committed as machine-readable JSON
policy and human-approval controls in the execution path
optional memory integration hooks (including supe) without hard coupling

Features

Versioned model/tool primitives: computer_20250124, computer_20251124
Typed action schema: screenshot, mouse_move, left_click, type, key, scroll, left_click_drag, wait, zoom
Runtime abstractions: EnvRuntime, EnvSession, ComputerToolRuntime, PolicyRuntime, MemoryAdapter, HumanApprovalRuntime
Native backend (NativeEnvRuntime) with Simulated (deterministic) and MacOsCoreGraphics (real macOS capture/input) modes
Browser backend (PlaywrightEnvRuntime) via scripts/playwright_action.mjs
Agent loop scaffold and model adapter interface with Anthropic implementation
Conformance utility module and report generator

Architecture

Model Loop
  -> Tool Runtime (validation + version checks)
      -> Policy Runtime (allow/deny/require-human)
          -> Human Approval Runtime (optional callback)
              -> Environment Backend (Native | Playwright | Mock)
  -> Memory Adapter (optional recall + action/summary logging)

Quickstart

cargo test -q
cargo xtask conformance
cargo xtask demo

Artifacts are written under demo-outputs/.

Demo Suite

Example	Purpose	Primary artifact
`foundation_demo`	Minimal loop and runtime wiring	`demo-outputs/foundation_demo.txt`
`native_backend_smoke`	Basic native backend action execution	`demo-outputs/native_backend_smoke.txt`
`native_real_world_demo`	8-step native trace and summary	`demo-outputs/native_real_world_demo_trace.json`
`playwright_backend_smoke`	Browser backend smoke path	`demo-outputs/playwright_backend_smoke.txt`
`action_showcase`	Action coverage and version behavior	`demo-outputs/action_showcase.txt`
`policy_guardrails`	Policy deny + require-human flows	`demo-outputs/policy_guardrails.txt`
`loop_trace_demo`	Multi-step transcripted loop	`demo-outputs/loop_trace_demo.txt`
`conformance_report`	Backend parity report generation	`demo-outputs/conformance/comparison.json`

Run all demos:

./scripts/run_oss_demos.sh

Conformance Artifacts

cargo xtask conformance generates:

demo-outputs/conformance/native.json
demo-outputs/conformance/mock.json
demo-outputs/conformance/comparison.json

Expected green signal in comparison.json:

"shape_matches": true
"differences": []

Native macOS Real Mode

Simulation is default. To run real machine input/capture on macOS:

NATIVE_DEMO_MODE=macos cargo run --example native_real_world_demo

Programmatic configuration:

use computer_use_runtime::backends::native::{NativeDriverMode, NativeEnvRuntime};

let env = NativeEnvRuntime::new()
    .with_driver_mode(NativeDriverMode::MacOsCoreGraphics)
    .with_sleep_on_wait(true);

Requires macOS Accessibility and Screen Recording permissions.

Live Anthropic + Playwright + Supe

Setup:

./scripts/setup_playwright_backend.sh

Environment:

export ANTHROPIC_API_KEY=...
export SUPE_PROJECT=computer-use-runtime
export ANTHROPIC_MODEL=claude-sonnet-4-5-20250929

Run:

cargo run --example live_anthropic_playwright

If browser launch is restricted in your environment, the runtime returns detailed fallback diagnostics.

Project Layout

src/model.rs: protocol models and tool_result payloads
src/runtime.rs: environment runtime/session traits
src/tool_runtime.rs: validation, policy, approval, and dispatch
src/policy.rs: policy interfaces and default policies
src/memory.rs: memory adapter contract and supe adapter
src/agent.rs: loop runtime scaffold
src/anthropic.rs: Anthropic loop model adapter
src/backends/native.rs: native backend + driver split
src/backends/playwright.rs: Playwright backend
src/conformance.rs: conformance and report utilities
examples/: runnable demonstrations
xtask/: contributor automation commands

OSS Demo Guide

For a release-grade walkthrough and expected outputs, see docs/OSS_DEMO_GUIDE.md.

Contributing

Contributor process, validation requirements, and reporting guidance are in CONTRIBUTING.md.

Release and Repo Bootstrap

release/distribution playbook: RELEASE.md
git + GitHub CLI bootstrap flow: docs/REPO_BOOTSTRAP.md
bootstrap helper script: scripts/bootstrap_repo.sh

Repo Launch Checklist

Verify badge/repo links match xayhemLLC/computer-use-runtime.
Verify LICENSE and Cargo.toml license metadata remain aligned.
Push main and verify Conformance workflow passes.
Attach demo-outputs/conformance/*.json in first release notes.