In one sentence
An AI agent that operates software through a screen, mouse and keyboard, the way a person would, instead of through an API.
A computer use agent is an AI system that looks at screenshots of a desktop or browser, decides where to click and what to type, and repeats until a task is done. Anthropic offers a computer use tool for Claude and OpenAI offers a computer use tool in its API. Both work on the same loop: the model sees the screen, requests an action, your code carries it out, and a new screenshot goes back to the model.
Where it helps
Some business systems have no API, or the API does not cover the screen people use every day. Old practice management systems, carrier portals and internal tools are common examples. A computer use agent can work through those screens without a custom integration.
Where it struggles
Computer use is slower and less predictable than an API call. Small layout changes, pop-ups and slow pages can throw it off. It costs more per task because every step sends an image to the model. Both vendors' documentation recommend running it in an isolated environment, such as a virtual machine or container with limited access, and keeping a person involved for consequential actions.
Security matters more here
An agent that can click anything a user can click has wide reach. Content shown on screen can carry prompt injection. Use a dedicated machine, a dedicated account with only the permissions the task needs, no saved passwords for unrelated systems, and human approval before submitting, paying or deleting. Check each site's terms too, since some portals prohibit automated access.
How we decide
We use an API or a database connection when one exists, because it is faster, cheaper and easier to test. We use computer use for the remaining screens, in a sandbox, with logs of every action. Our post comparing computer use costs for invoice entry shows the trade-off in numbers.
Keep following the system
AI Agent
A system that uses a model and tools to pursue a goal across multiple steps within defined permissions and review boundaries.
Tool Calling (Function Calling)
A model feature where the AI returns a structured request to run a defined function, which your code runs and returns.
Excessive Agency
The risk that an AI system can take damaging actions because it has more functions, permissions or autonomy than its task needs.
Webhook
An HTTP message one system sends to another the moment an event happens, so the receiver does not have to keep asking.
Idempotency
The property that doing the same operation twice has the same effect as doing it once, so retries do not create duplicates.