Computer-use agents re-read the whole screen every turn. A binary change mask that marks only the pixels that moved.
Your AI agent can’t click what it can’t see. But it also shouldn’t have to re-read the entire screen every time it takes a step.
That’s the loop every computer-use agent runs: screenshot the screen, send it to the model as vision input, get back a click or a keystroke, execute it, screenshot again. Anthropic’s computer use toolset ships seventeen member tools — screenshot, left_click, type, zoom — and the docs are explicit that managing screenshot history is part of building a working agent, not an afterthought.
Here’s the thing that surprised me when I went looking for numbers. Screenshots are not the expensive part. A measurement of 25,457 real agent tool results across 329 sessions found a screenshot costs a median of 2,054 tokens — essentially the same as reading a single file (2,085). Screenshots were the most predictable tool in the entire set, varying only 1.7× between median and p90, because an image is billed by pixel area and a browser window is always roughly the same area.
So the problem isn’t cost. It’s attention. Most of the screen is identical between steps. Only a small region changed: a menu opened, a toast appeared, a button toggled. You’re paying to re-describe a static desktop, and the model has to find the one…