This post is about background desktop control for any MCP-capable agent: one targeted window, clicks and typing without stealing focus, over MCP, CLI, or SDK. Hermes ships the same driver as its built-in `computer_use` toolset. That is different from LUCI (recall-only screen memory), Google Home MCP (physical devices in a structure), Claude Code's session-bound Mac `computer-use` server, Cursor Self-Hosted (cloud planning, your worker executes), or Simular's vendor-operated GUI fleet.
Cua Driver is open source under MIT in the trycua/cua repository. It does not run a model; your agent supplies the loop. This post stays with Cua's product page, how the driver works, connecting an agent, permissions, platform support, and Hermes Computer Use docs. The newest release we checked is 0.34.0, published October 5, 2026 on GitHub and PyPI.
Short answer
Use Cua Driver when an agent needs a desktop app that has no API or MCP tool, and the person at that computer should be able to keep working. Before anything runs unattended, start it with a narrow permission manifest. If you run Hermes, you already have it: Hermes ships Cua Driver as its built-in `computer_use` toolset. If the work lives in a web app or behind an API, give the agent a browser tool or a typed MCP tool first.
What Cua Driver does
It runs on macOS 14 and later, Windows 10 and 11 (and Windows Server), and x86_64 Linux desktops on X11 or XWayland with AT-SPI 2. Native Wayland works with limits that depend on the compositor.
Each step has two halves. To read the screen, the agent calls `get_window_state` with a process id and window id and gets back that window's accessibility tree and a screenshot in one response. To act, it calls a tool such as `click`, `type_text`, `press_key`, `scroll`, or `drag`, and every call names the window it targets. Input goes to that window in the background by default. The driver does not raise the window, move the real pointer, or switch the front app. The agent gets its own overlay cursor so you can see where it is working.
The same binary runs three ways:
- `cua-driver mcp` starts an MCP server over stdio for any client that can spawn a process.
- `cua-driver serve` keeps a daemon running so sessions and window state last between calls.
- `cua-driver call` runs one tool call from the shell, for scripts and debugging.
Cua's page lists Hermes, Clicky, H Company, and Factory Droid as products that use it for computer use. Cua also offers hosted desktops through Cloud Fleets and a Sandbox SDK. This post covers only the driver.
Background first, foreground when the app needs it
Some apps do not accept background input. Cua documents a fixed order of attempts, which it calls the action ladder:
- 1Act on an accessibility element in the background. This is the only route the driver can verify by itself, by reading the change back.
- 2Act on pixel coordinates from the same screenshot, still in the background.
- 3For browser tabs, switch to the browser tools, which work on the page over CDP without focus.
- 4Retry only that one action in the foreground. The window comes forward, the input lands, and the previous app is restored. Cua names games, canvas apps such as Blender, and apps that poll for focus as cases that need this.
Every response reports what happened. The `effect` field reads `confirmed`, `unverifiable`, `suspected_noop`, `partial`, or `refused`, and an `escalation` field suggests the next rung when one exists. Cua warns that a delivered event does not prove the app changed: Electron, Catalyst, and web content can echo a write they did not apply. After any result short of `confirmed`, the agent should take a fresh snapshot and check. Where the operating system cannot deliver safe background input, such as raw keys to another window on standard Wayland, the driver refuses instead of risking input to the wrong app.
All sessions on one computer share the same screen, keyboard, and focus, and the driver admits native input one action at a time. Two agents can connect at once, but they take turns on the desktop.
Permissions are fixed at startup
Every call from the CLI, MCP, the SDK, or the daemon passes the same check inside the driver. The launcher sets the mode at startup. An agent cannot widen it from a tool call, and changing it means restarting the daemon.
| Mode | What it allows | Use it for |
|---|---|---|
| `standard` (default) | Observing, clicking, typing, isolated browsers, recording, and file transfer to validated paths, all without prompts. Attaching to a logged-in browser profile needs a grant. | Local use with a person nearby |
| `bounded` | Only the tools and resources listed in a capability manifest. Everything else is denied. | Unattended agents, gateways, embedded apps |
| `unrestricted` | Skips Cua's approval checks. Cua says this gives no protection against prompt injection. | Disposable or fully trusted machines |
The manifest is a YAML file that denies by default. It lists the allowed tools, the apps the agent may launch and touch, the folders it may read or write, and whether full-display capture is allowed. In `bounded` mode it also needs an expiry and an idle timeout. You can add YAML or Rego policies that limit tool arguments, such as a maximum length for typed text.
The docs flag two gaps to plan around. Registering Cua Driver in a client does not choose a permission mode, and the default `standard` mode allows input to every app. Policies also do not rate-limit or authenticate callers, and other code running as the same user on that desktop is outside the boundary.
How it differs from Claude Code computer use
Our Claude Code computer use post covers Anthropic's built-in `computer-use` MCP server. Both put screen control behind MCP. They make different choices about who else is using the screen.
| Question | Claude Code computer use | Cua Driver |
|---|---|---|
| Which agents can use it? | Claude Code | Any MCP client or shell-capable agent, Claude Code included |
| Platforms | CLI on macOS; Desktop app on macOS and Windows | macOS, Windows, Linux |
| Your screen while it works | Apps you did not approve are hidden until the turn ends (always in the CLI) | Input goes to one window in the background where the app allows it; your cursor and focus stay put |
| How access is granted | You approve each app per session, with fixed control tiers | A mode and optional manifest set at launch |
| More than one session | One session holds a screen lock | Several sessions can connect; they share one screen and take turns on input |
| Cost and license | Part of Claude Code on a Pro or Max plan | MIT open source |
Claude Code's server fits a developer on a Mac who wants the agent to click through the app it just built while they watch. Cua Driver fits when the agent is something other than Claude Code, when the machine runs Linux, or when the person at the keyboard needs to keep working.
How it differs from Muse and Simular-class agents
Muse Secure VM and Simular Sai are full agent products. The vendor supplies the agent and the computers it works on: a cloud VM per user for Muse, a fleet of computers for Sai. Cua Driver is one layer of that stack. Cua's docs say the harness owns the model and the loop and the driver never calls a model API. You bring the agent, the model, and the machine, so the trust boundary is yours to set.
Hermes already runs on it
Hermes Agent's built-in `computer_use` toolset talks MCP over stdio to Cua Driver. Hermes pins the driver version and installs it by default on macOS, Windows, and glibc Linux. Cua's setup guide says to turn it on with `hermes tools enable computer_use --platform cli`, start a session with `hermes -t computer_use chat`, and not add a second raw MCP server on top.
Hermes adds its own controls around the driver:
- Clicks, typing, drags, scrolls, key presses, and focus changes go through the same approval gate as dangerous shell commands.
- Where nobody can answer, as in cron or headless runs, those actions are refused instead of auto-approved, unless you turn on `--yolo`.
- Default and smart approvals run Cua in `standard` mode. A reviewed manifest runs a private `bounded` runtime. YOLO runs `unrestricted`, which the Hermes docs say to use only in a disposable VM or with accounts and data whose full compromise you accept.
- Hermes turns off Cua Driver's anonymous telemetry, which is on by default upstream, unless you opt back in.
- With a sandboxed terminal such as Docker or SSH, the driver runs inside the sandbox on that sandbox's display, so it can only touch what the terminal can reach.
The Hermes docs also point web-only tasks to the `browser` toolset, which uses headless Chromium. They also note that on macOS the driver uses private SkyLight interfaces, which Apple can change in any OS update.
For OpenClaw, `cua-driver mcp-config --client openclaw` prints an `openclaw mcp set` command to run. On macOS the Accessibility and Screen Recording grants belong to CuaDriver.app, and Cua notes that a server spawned by the OpenClaw gateway does not inherit OpenClaw.app's grants.
Setting it up on a Mac
- 1Install with `/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"`. Cua says no administrator access is needed.
- 2Start the daemon through the app bundle so macOS attributes the permission prompts to CuaDriver: `open -n -g -a CuaDriver --args serve`.
- 3Run `cua-driver permissions grant` and switch CuaDriver on under Accessibility and Screen & System Audio Recording.
- 4Check that it can see your apps with `cua-driver call list_apps`. An empty list means it cannot reach the desktop, and `cua-driver doctor` reports why.
- 5Register it with your agent. For Claude Code, run `claude mcp add --transport stdio cua-driver -- cua-driver mcp`. For other clients, `cua-driver mcp-config --client <client>` prints the exact command or JSON.
On Linux, run `cua-driver serve` in a terminal inside the desktop session and leave it open. On Windows, install with the PowerShell script and run `cua-driver autostart kick` from the desktop session. A Windows SSH login lands in Session 0, which cannot see the desktop, so driving Windows over SSH needs Cua's autostart setup.
Before an unattended run, write a manifest and start the daemon with `cua-driver serve --permission-mode bounded --capability-manifest ~/cua-session.yaml --approve-capability-manifest`. Then test it the way the docs describe: an allowed call succeeds, the same tool against another app or path returns `bounded_resource_outside_manifest`, and a tool missing from the list returns `permission_denied`.
When to use it
Use Cua Driver when:
- A step in the workflow needs a desktop app with no API or MCP server, such as a CAD drawing, an iPhone Simulator, or an older desktop tool your team still depends on.
- Someone needs to keep using the same computer while the agent works.
- You run more than one agent harness and want them on one driver with one permission setup.
- The computer runs Linux. Claude Code's built-in server covers macOS and, in the Desktop app, Windows.
Reach for something else first when:
- The system has an API or MCP tool. A typed call is easier to check, log, and retry than a click on a screenshot, as we covered in when an agent uses the browser instead of the MCP tool.
- The work is in a web app. A browser tool, or Cua's own page tools, fits better than pixel clicks.
If the computer is signed in to accounts that matter and nobody will be watching, run `bounded` with a short expiry or keep the job off that machine.
How this fits implementation work
OrchestriAI does not sell or operate Cua Driver. When a client workflow depends on a desktop app, MCP development checks first whether a typed tool can replace the screen. AI agent systems decide which steps an agent may take and where a person approves. If you run Hermes or OpenClaw, Hermes Agent setup and OpenClaw setup cover the host, approvals, and tool policy that computer use sits under. For the wider trust picture, see trust boundaries for self-hosted agents and permissions and human approval.
Independence and limitations
OrchestriAI is an independent implementation provider. Product names are used for identification only. OrchestriAI is not affiliated with, endorsed by, or sponsored by Cua, Anthropic, Nous Research / Hermes, OpenClaw, Meta, or Simular. This article summarizes Cua's public Cua Driver page and docs, the trycua/cua repository README and 0.34.0 release notes, and the Hermes Computer Use docs as checked on October 7, 2026. We have not benchmarked Cua Driver, and the products Cua lists as users are Cua's own statement. Cua shipped eight PyPI releases between September 30 and October 5, 2026, so commands and limits can change quickly. Confirm the current docs before you give it a machine that holds client data. This is not a security certification or a compliance conclusion.
