GitHub - lgxz/winuse: App Control for Agents

Native GUI automation with a Rust core, a Python API, and an agent-oriented CLI.

winuse supports macOS Accessibility and Windows UI Automation behind the same Backend trait.

Development

On macOS, winuse requires macOS 15.2 or later. Grant Accessibility permission to the terminal or Python host that runs winuse. Screenshots additionally require Screen Recording permission.

On Windows 10 or 11, UI Automation and visible-region screenshots need no separate privacy permission. Windows integrity isolation (UIPI) can block input to an elevated application, so run winuse at the same integrity level as the target. Secure desktop windows are not supported.

python3 -m venv .venv
.venv/bin/pip install maturin pytest
.venv/bin/maturin develop
.venv/bin/pytest

Python API

Move text through the system clipboard (pair with copy/paste shortcuts):

desktop.clipboard_write("Hello")                 # then press cmd+v to paste
clip = desktop.clipboard_read()
print(clip.kind, clip.text, clip.files)          # text | files | image | empty
desktop.clipboard_read(save_image="shot.png")    # export an image clipboard

Control windows by id:

window = desktop.window(app="TextEdit")
window.activate(); window.minimize(); window.restore(); window.maximize(); window.close()
desktop.windows()           # all real windows; each has .minimized

Wait for an async UI state instead of polling snapshots yourself:

send = window.find(role="button", name="Send")
if send.wait_until("enabled", timeout=10):
    send.click()
window.find(role="static-text", name="Status").wait_until("text", text="Done", timeout=30)

from winuse import Desktop

desktop = Desktop()
if not desktop.is_authorized():
    raise RuntimeError("The platform accessibility backend is unavailable")

window = desktop.window(app="TextEdit")
editor = window.find(role="text-area")
editor.fill("Hello from winuse")
editor.press("enter")
editor.fill("query", submit=True)                 # fill then press Enter (search boxes)
editor.type_text("inserted at the cursor")      # type vs fill: inserts, does not replace
editor.press("s", modifiers=["cmd"])             # shortcut; cmd is Ctrl on Windows
editor.click(double=True)                        # double-click
editor.hover()                                   # hover without clicking
window.find(role="list-item", name="File").drag_to(name="Folder")  # drag onto a target

subtree = window.find(role="group", name="Sidebar").snapshot(max_depth=5)
print(subtree.text())

Watch a subtree for semantic additions and removals:

messages = window.find(role="list", name="Messages")
with messages.watch(interval=0.25) as changes:
    while change := changes.next(timeout=30):
        print(change.kind, change.element)

Inspect every accessibility attribute exposed by an element:

inspection = window.find(role="static-text", name="Message").inspect()
print(inspection.text())

Full native-attribute inspection works on both platforms: macOS reports the element's AX attributes, Windows reports the UIA properties the element supports (name, automation id, class name, value, patterns, …).

Capture the visible screen region occupied by an element:

button = window.find(role="button", name="Send")
png = button.screenshot("send-button.png", padding=8)

The method always returns the PNG bytes. Passing a path also writes the same bytes to that file. The captured region reflects what is currently visible on screen, including any windows covering the element.

Elements store a selector, not a long-lived native handle. Each action resolves the element against a fresh accessibility snapshot.

CLI

Build the CLI:

cargo build --release -p winuse-cli

Install the agent skill (SKILL.md + references + host binary) to ~/.agents/skills/winuse, or package a distributable zip with all three platform binaries:

make install                  # DEST=... to override the location
make package                  # → target/dist/winuse-<version>-skill.zip

Commands accept ordinary flags and always return a JSON envelope:

target/release/winuse doctor
target/release/winuse windows
target/release/winuse snapshot --window 123
target/release/winuse click --window 123 --role button --name Send --exact
target/release/winuse fill --window 123 --role text-area --text "Hello"
target/release/winuse press --window 123 --key enter
target/release/winuse scroll --window 123 --direction up --amount 500

The repository also contains a distributable agent skill at skills/winuse. Its bundled executable is built from winuse-cli.

Acknowledgments

winuse was inspired by and references agent-browser (Vercel Labs) — its approach of driving a UI semantically for agents, with stable selectors and JSON tool responses, shaped winuse's design for native desktop apps.