Understanding Computer Use
What is Computer Use?
Computer Use is Anthropic's production API that allows Claude to control computers through visual interface interaction. Unlike traditional AI integrations that require specific APIs or custom code for each application, Computer Use enables Claude to interact with any software the same way a human would.
The trade you are actually making
Computer Use is not strictly better than scripted automation — it trades determinism for reach. Pick it when there is no API to call, not when there is one you'd rather not read the docs for.
Scripted automation vs. Computer Use
Scripted (API / Selenium / Playwright)
- Same input gives the same output, every time
- Cheap enough to run on every commit
- Failures are loud - a selector throws
- One integration per service, written and maintained by you
- A renamed CSS class breaks it silently in production
- Useless where there is no API and no stable DOM - legacy desktop apps, canvas UIs, PDFs in a viewer
Computer Use
- Works against any pixels - no API, no DOM, no vendor cooperation
- Survives cosmetic UI changes that break selectors
- One agent generalises across applications
- Non-deterministic: the same task can take a different path each run
- Costs real money per step, and context grows as screenshots accumulate
- Failures are quiet - it can do the wrong thing confidently and report success
- Anything on screen is reachable, which is why sandboxing is not optional
The mechanism underneath is simple: Claude sees the screen through screenshots, understands it with vision, and acts by returning mouse and keyboard calls that your code carries out.
Key Capabilities
The Computer Use API provides Claude with these tools:
| Tool | Purpose |
|---|---|
computer | Take screenshots, move mouse, click, type, scroll |
text_editor | View and edit files directly |
bash | Execute terminal commands |
Reading the benchmark numbers (and why we don't print them here)
The benchmark you will see quoted for computer use is OSWorld, which scores a model on real tasks in a live desktop — browsers, terminals, LibreOffice — driven by synthetic clicks and keystrokes.
This course deliberately does not print a score, because that number is less stable than it looks. Three things move underneath it:
- The benchmark gets revised in place. OSWorld-Verified is an in-place upgrade of the original OSWorld — same family, new task quality, grading, and infrastructure. Anthropic's own note is explicit that scores before Sonnet 4.5 were measured on original OSWorld and scores from 4.5 onward use OSWorld-Verified, so a chart that runs one line across both is comparing two different rulers.
- Scores get restated for models that already shipped. A model's published figure can move after a methodology change, without the model changing at all.
- The frontier moves every few months. Any "model X leads" sentence has a shelf life measured in weeks.
So the durable skill is not memorising a percentage — it is knowing how to read one:
| When you see… | Ask |
|---|---|
| A single headline score | Which benchmark revision? Verified, 2.0, or original? |
| A trend line across model generations | Were all points measured on the same revision? |
| "Human baseline" | Published by whom, on which task subset? |
| "State of the art" | Dated when? |
For the current numbers, read Anthropic's model overview and the launch post for whichever model you deploy — those are the only sources that get corrected when a score is restated.
What the scores do tell you, directionally: computer use crossed from "demo" to "useful" across recent model generations, and it is now good enough that the interesting failures are no longer "it clicked the wrong button" but "it did the wrong task confidently." That is why Modules 3–5 spend more time on guardrails than on clicking.
Note: Computer Use is enabled via the
computer-use-2025-11-24anthropic-beta header and uses thecomputer_20251124tool type. The header signals a versioned, opt-in surface — not immaturity. Run agents in sandboxed environments (Docker, VM, or a dedicated browser profile) regardless: the agent sees, and can act on, whatever your screen shows.
What You'll Build
Capstone preview — the Job-Form Filler. Every technique you learn across the next five modules stacks into this one real agent:
Module 1 → mental model of screenshot → vision → action
Module 2 → Docker sandbox + Agent SDK setup
Module 3 → Desktop automation (the window-opening half)
Module 4 → Browser automation (forms, auth, session handling)
Module 5 → Production safety (dangerous-action guardrails, audit logs)
Capstone → All of the above, wired together for real applications
Build checkpoint — do this before the next lesson
- Collect 3–5 real job descriptions you might actually apply to. Save as
.txtfiles in ajds/folder. These are your capstone inputs. - Find 2 job-board form URLs you want the agent to fill. Public application forms (Greenhouse, Lever, personal career sites) work well. Private boards requiring login need Module 4's auth patterns.
- Prepare your resume as a PDF at a known path. The capstone's file-upload tool needs an absolute path.
- Install Docker Desktop if you haven't — Module 2 runs the agent in a sandboxed container so it can't damage your host system.
In the next lesson, we'll explore how Computer Use works under the hood. :::
Sign in to rate