Your Agent Doesn't Need to See the Screen—It Needs to Read It

If you've ever tried to get an AI agent to click a button in a desktop app, you know the pain. Screenshot-based approaches work until they don't: a font renders slightly differently, a theme changes, a dialog shifts ten pixels, and suddenly your agent is clicking into the void. You end up babysitting a process that's supposed to be autonomous, wondering why it can't just ask the app what's on screen instead of squinting at pixels.

That's the problem agent-desktop is built to solve. It's a Rust-based tool that gives agents reliable computer use on the desktop by reading OS accessibility trees—the same structured data that screen readers use—rather than guessing from images.

What It Does

agent-desktop is a native Rust CLI (compiled as a single binary with no runtime dependencies) that observes, decides, and acts on desktop applications. Instead of taking screenshots and running them through vision models, it queries the operating system's accessibility APIs to get the real UI structure of any app. That means it can see buttons, menus, text fields, and windows as structured elements with stable references—not as coordinates on a bitmap.

The tool exposes 58 command names covering observation, interaction, keyboard and mouse input, notifications, clipboard, window management, session lifecycle, and trace export, plus a bundled skills doc loader. Four of those names are reserved for a stateful daemon and fail closed in the stateless CLI. It's designed to work with anything that has an accessibility tree: Finder, Safari, System Settings, Xcode, Slack, you name it. There's also a C-ABI cdylib (libagent_desktop_ffi) so you can load it once from Python, Swift, Go, Ruby, Node, or C instead of forking the CLI on every call.

For web content inside Chromium apps, there's a launch --cdp flag that opens a verified DevTools port, letting frameworks like Playwright, Puppeteer, or chrome-remote-interface drive the web contents while native menus and dialogs stay on the accessibility path.

Why It's Cool

The core insight here is simple but powerful: accessibility trees give you structure, and structure gives you reliability. When your agent references @s8f3k2p9:e1, it's pointing at a specific element in a specific snapshot—not a coordinate that might drift. Refs stay stable, actions stay safe to retry, and you're not burning tokens on screenshots.

• Progressive skeleton traversal is the standout feature. Dense apps like Slack can produce accessibility snapshots that are enormous—the README shows an example where a regular snapshot weighs in at 30,743 tokens. A skeleton overview of the same app comes in at 383 tokens. That's a 78–96% token reduction on dense applications, achieved by starting with a shallow overview and drilling down only into the controls you actually need. If you're paying per token, this matters enormously.

• Headless-by-default interactions are a quiet win. Ref actions use accessibility APIs and explicitly block silent focus, cursor, keyboard, or pasteboard side effects. Your agent can interact with an app without hijacking your mouse or clipboard in the background—which means you can actually use your computer while it works.

• Structured JSON output with error codes and recovery hints. No parsing HTML or scraping logs. You get machine-readable responses that tell you what happened and, when things go wrong, what to do about it.

• One binary, many languages. The C-ABI cdylib means you're not locked into a specific runtime. If you've got an existing Python automation stack or a Swift app, you can load the library directly rather than shelling out to a CLI process for every action.

How to Try It

The recommended install path is npm, which downloads a prebuilt binary:

npm install -g agent-desktop

Once installed, you can start exploring the command set. The CLI exposes 54 operational commands spanning observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle, and trace read/export. There's also a bundled skills doc loader to help you get oriented.

For Chromium-based apps where you want to drive web content through CDP:

agent-desktop launch --cdp

That opens a verified DevTools port so any CDP-speaking framework can take over the web contents while native UI stays on the accessibility path.

The project is on GitHub at github.com/lahfir/agent-desktop, licensed under Apache-2.0. You'll find architecture diagrams and a Slack example showing the token difference between full snapshots and skeleton overviews in the repo.

Final Thoughts

agent-desktop is a focused tool for a specific problem: making desktop automation reliable enough that agents can actually be trusted to operate apps without constant supervision. The accessibility-tree approach isn't new—screen readers have used it for decades—but applying it to agent workflows, with token-efficient traversal and a clean CLI interface, is a practical combination. If you're building agents that need to interact with desktop software, or if you've been frustrated by the brittleness of pixel-based approaches, this is worth a look. The token reduction numbers alone make a strong case for trying the skeleton traversal workflow before you reach for another screenshot pipeline.

Follow @githubprojects for more developer tools and open source projects.