AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Why AI Agents Click the Wrong Button: New Benchmark Traces Failures to Spatial Reasoning

Computer-use agents that operate software by reading screenshots and emitting coordinates are entering real products. Yet when an instruction says “click the button above Save” or “the third item in the menu,” current vision-language models often miss. A new diagnostic benchmark, GUI-PRIMITIVES, aims to pinpoint which of seven elementary spatial skills breaks in screenshots.

The benchmark, built by researchers at USC and Qatar Computing Research Institute, contains 994 contrastive-pair items. Each pair holds the screenshot and an anchor element fixed but changes a relational expression—“right of” becomes “left of,” “inside” becomes “outside”—so the correct target shifts between two designated candidates. This blocks shortcuts: a model that always clicks the most salient element passes one twin and fails the other. Five annotators validated a 196-item subset, with high agreement (Fleiss Îș=0.94 well-formedness; Îș=0.79 target selection).

In one example, four models—Claude Opus 4.7, GPT-5, Qwen2.5-VL-7B-Instruct, and OS-Atlas-Base-7B—got paired instructions about a “Reset” element: one said “currently fully visible,” the other “currently partly hidden.” On both versions, the models clicked the partly hidden button, the authors report, drawn to the visually unusual element instead of the relation.

Across 19 vision-language models, the strongest, Claude Opus 4.7, achieved 32.4% strict point-in-box accuracy (predicted clicks within the target’s box) on the 185-item human-clean subset; humans selected the correct target on 96.9% of those items, a comparison the paper says is imperfect. The top five models scored 0.80–0.83 on list-ordinal questions, but on containment, occlusion, alignment, and proximity, every model fell below the 0.50 two-candidate reference level.

Classifying predictions by candidate region showed that on 60–92% of items, predictions fell outside both candidate regions; on real desktop screenshots from UI-Vision, the share was 96.2%. When they did land in a candidate region, models picked the target on 0.89–0.90 of horizontal, vertical, and proximity items. For containment and occlusion, that rate was statistically indistinguishable from chance. The authors read this as a localization problem, not a relation-language problem—except for containment and occlusion, where no measurable relation signal appeared.

Scores also correlated with ScreenSpot-Pro, a standard grounding test, across ten models (Spearman ρ=+0.74, p=0.015), an association the authors call exploratory. A training-free intervention, Set-of-Mark prompting (numbering the two candidate elements), raised selection accuracy by 35–57 percentage points—GPT-5 from 30% to 87%, OS-Atlas from 10% to 52%. That intervention supplies the candidate set the task asks models to find, so the authors call it a diagnostic upper bound, not a deployable fix. Chain-of-thought prompting and activation steering, which adjusts internal model states, produced no significant gain; the authors note that chain-of-thought models often named the relation correctly but still clicked the wrong spot.

A limitation: containment and occlusion appear only in synthetic screenshots, because real GUI datasets lack the needed metadata; findings on those primitives may not fully transfer to real interfaces. Still, the work offers a way to separate “can’t find it” from “doesn’t understand the instruction,” a distinction that could guide targeted fixes before agents are trusted with real clicks.