GitHub - varbhat/desktopmcp: MCP server for the Linux desktop 🐧🤖

MCP server for the Linux desktop. It gives AI models access to the Linux desktop.

desktopmcp connects AI assistants to the Linux desktop through three system interfaces: XDG Desktop Portals for sandboxed desktop operations, AT-SPI for semantic UI understanding, and D-Bus for low-level system access. It exposes 144 tools over the Model Context Protocol, letting an AI see what's on screen, understand UI structure, move cursor, click buttons, type text, manage files, and interact with desktop services.

Why

AI models are blind to the desktop. They can't see a window, read a button label, click a menu item, or control the desktop.

desktopmcp solves this by giving AI models two complementary ways to interact with the desktop:

• Visual: screenshots, screen capture, color picking, mouse and keyboard input
• Semantic: the full accessibility tree (AT-SPI), where every UI element has a name, role, position, and set of actions the AI can invoke directly

The semantic path means an AI can call find_element(role="push button", name="Save") instead of scanning pixels to find where "Save" might be. It can read text content, check which element is focused, traverse the UI tree, and perform actions like "click" on any element -- all without needing a screenshot.

Everything goes through XDG Desktop Portals, so the user gets permission dialogs, and a single binary runs on GNOME, KDE, Sway, or any Wayland compositor.

What it can do

144 tools organized into five groups:

Architecture

AI model (Claude, etc.)
    |
    |  MCP protocol (JSON-RPC over stdio or HTTP)
    v
desktopmcp               (Rust, async, single binary)
    |
    |--- XDG Desktop Portals  (ashpd)  --> screenshots, input, files, settings, ...
    |--- AT-SPI bus            (zbus)   --> UI tree, element actions, text, events
    |--- D-Bus session/system  (zbus)   --> arbitrary service access
    |--- PipeWire              (pipewire-rs) --> screen capture frames
    v
Linux desktop  (GNOME / KDE / Sway / ...)

XDG Portals ensure every sensitive operation (screen capture, input injection, file access) goes through a user-facing permission dialog. The AI cannot act without the user granting consent.

Requirements

• Linux with a Wayland compositor (GNOME 40+, KDE Plasma 5.27+, Sway, or similar)
• xdg-desktop-portal and a desktop-specific backend (xdg-desktop-portal-gnome, -kde, -wlr)
• PipeWire (for screen capture)
• AT-SPI2 (at-spi2-core, typically pre-installed on any desktop with accessibility support)
• D-Bus session bus (present on all modern Linux desktops)

Build requirements:

• Rust 1.85+ (edition 2024)
• System libraries: libdbus, libpipewire-0.3, at-spi2-core, pkg-config
• LLVM/Clang (for pipewire-sys bindgen)

Installation

Using Nix (recommended)

Run directly without cloning:

nix run github:varbhat/desktopmcp -- --t http # or stdio

Or clone and build locally. The repository includes a flake.nix that provides all system dependencies automatically:

git clone <repo-url>
cd desktopmcp
nix develop
cargo build --release

Release AppImage (recommended)

We use Github Actions to build and release AppImages automatically. Grab the latest AppImage from the releases page. Mark it as executable and run it. It's built using nix-appimage which bundles the nix derivation and all its dependencies into a single-file executable, and hence the size of the AppImage is very big (It's something we'll optimize in upcoming releases—but hey, it lets you try it!).

Without Nix

Install system dependencies, then build:

# Debian / Ubuntu
sudo apt install libdbus-1-dev libpipewire-0.3-dev at-spi2-core \
                 pkg-config libclang-dev

# Fedora
sudo dnf install dbus-devel pipewire-devel at-spi2-core-devel \
                 pkg-config clang-devel

# Arch
sudo pacman -S dbus pipewire at-spi2-core pkgconf clang

# Build
cargo build --release

The binary is at target/release/desktopmcp.

Usage

# stdio mode (default; Your MCP Client will do this for you)
desktopmcp

# HTTP mode -- for remote-mcp (You need to run this beforehand)
desktopmcp --transport http --bind 127.0.0.1:3000

In stdio mode, MCP messages are exchanged over stdin/stdout (JSON-RPC). Logs go to stderr. In HTTP mode, the server listens at http://<bind>/mcp using Streamable HTTP with Server-Sent Events. You can configure your MCP Client to use desktopmcp in stdio mode by specifying the binary path. Or you can run the desktopmcp in HTTP remote-mcp mode and configure your MCP Client to use it by specifying the URL.

Quick examples

Ask AI: "What windows are open on my desktop?"

The AI calls get_window_list and gets back a structured list of every window with title, application name, position, and size -- no screenshot needed.

Ask AI: "Click the Save button in Firefox"

1. find_element(role="push button", name="Save", app_name="Firefox")
   -> returns element ID with position and available actions
2. atspi_do_action(id="...", action_name="click")
   -> button is clicked via the accessibility API

Ask AI: "Take a screenshot and tell me what you see"

1. start_session(devices=["pointer"], with_screencast=true)
   -> user approves the permission dialog once
2. take_screenshot(session_id="...", format="jpeg")
   -> AI receives a base64-encoded image

Ask AI: "Type my email address into the login form"

1. find_element(role="entry", name="Email")
   -> finds the text field
2. type_into(id="...", text="user@example.com")
   -> text is entered via the EditableText accessibility interface

Ask AI: "Send me a notification when the download finishes"

1. send_notification(id="dl-done", title="Download Complete", body="file.zip is ready")
   -> desktop notification appears

Tool reference

Remote Desktop & Input

These tools require an active session created with start_session. The user sees a one-time permission dialog.

XDG Portal Tools

These tools use sandboxed XDG Desktop Portal APIs. Most work without a session.

Dynamic Launcher

Install and manage .desktop application launchers.

AT-SPI Accessibility

Semantic access to every UI element on the desktop. No screenshot or coordinate guessing required.

Tree traversal & element inspection:

Search & UI tree:

Actions & interaction:

Text:

Component (geometry, scroll, focus):

Value, Selection, Table, Hyperlinks, Document, Image:

Events:

D-Bus Bridge

Direct access to any D-Bus service on the session or system bus. The AI can introspect and interact with arbitrary desktop services.

Development

nix develop              # enter dev shell with all dependencies
cargo build              # compile
cargo clippy             # lint
cargo run                # run in stdio mode
cargo run -- -t http     # run in HTTP mode

Verify portals are running:

systemctl --user status xdg-desktop-portal
systemctl --user status pipewire

Enable debug logging:

Test with a raw MCP message:

echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}' \
  | cargo run --quiet 2>/dev/null

Troubleshooting

Permission dialog does not appear

Restart the portal service:

systemctl --user restart xdg-desktop-portal

PipeWire connection fails

Check that PipeWire is running:

systemctl --user status pipewire
systemctl --user restart pipewire

AT-SPI returns no applications

Make sure the AT-SPI bus is running and accessibility is enabled:

# Check AT-SPI status
dbus-send --session --print-reply --dest=org.a11y.Bus \
  /org/a11y/bus org.freedesktop.DBus.Properties.GetAll \
  string:org.a11y.Status

# Enable accessibility (on Gnome) if disabled
gsettings set org.gnome.desktop.interface toolkit-accessibility true

Mouse/keyboard input has no effect

• Call start_session first with the required devices (["keyboard", "pointer"])
• Accept the permission dialog when it appears
• Verify the session is still active with a valid session_id

Security

desktopmcp is designed around the XDG Desktop Portal security model:

• Every sensitive operation requires explicit user consent via a system dialog
• The server works inside Flatpak and other sandboxed environments
• D-Bus and AT-SPI access follows standard Linux desktop permissions
• No root privileges are needed
• The user can deny or revoke access at any time

That said, this server gives AI models significant control over your desktop. Use it with trusted models, review what the AI does, and be aware that an unrestricted AI could perform unintended actions.

Technology

License

Apache License 2.0