AI Papers Reader

Personalized digests of latest AI research

View on GitHub

A new benchmark puts AI judges of computer-using agents to the test

Computer-using agents (CUAs) — AI systems that click, type, and operate browsers, phones, and desktops — increasingly depend on automated judges. After a run, a vision-language model (VLM) may decide whether the agent completed the instruction, a signal used for evaluation, data filtering, and reinforcement learning. But those judges themselves have never been systematically tested. A preprint led by researchers at the University of Hong Kong, Nanjing University, and other institutions introduces OSReward, a benchmark built to evaluate them.

Instead of reusing existing benchmark runs, the authors built fresh environments and collected 1,019 human-verified trajectories across web, mobile, Ubuntu, and Windows. Agents from the Claude, Gemini, Kimi, and Qwen families executed the tasks, and every trajectory received multi-stage human labeling, with disagreements escalated to senior reviewers. They also created OSReward-Hard, 284 deliberately deceptive cases, and OSReward-Multi for fine-grained quality ratings. They then ran 27 VLM judges under one shared protocol.

One web case captures the dominant failure mode. An agent asked to find recently popular documentaries with ratings above 8 selected a documentary visibly dated 2017 and presented it as recent. Human annotators marked the run a failure, but GPT-5.5 and Gemini-3.5-Flash accepted it. The authors found such over-acceptance widespread: accepting an incomplete task accounted for about two-thirds of judges’ errors.

On the full set, the best models looked strong: Claude-Opus-4-8 scored 89.7% binary accuracy and GPT-5.5 89.5%. On OSReward-Hard, performance collapsed. The same best judge fell to 69.7%, comparable to a constant always-fail baseline given the set’s 30/70 success/failure split, and the average judge fell to 52%. Even the steadiest frontier judges caught only about 70% of successes and failures on the hard set. Ablations suggested why: removing the agent’s per-step text cost judges 7.2 percentage points on average, while changing screenshot counts barely moved accuracy. The authors interpret this as judges relying on the agent’s narrative rather than verifying the screen.

Reliability is also expensive. A single modest reinforcement-learning run would need about 51,200 judge calls, the authors estimate: roughly $4,000 with Claude-Opus-4-8 or $2,300 with GPT-5.5, versus about $68 with OS-Shepherd-9B, an open reward model they trained—about a 30–60× reduction. To create it, they built OS-Shepherd-100K, an open corpus of about 100,000 reasoning-annotated trajectory judgments, keeping only trajectories on which diverse strong judges agreed. They trained a 9B and a 35B version; on OSReward-Hard, the 9B reached 60.2%, much better than similarly cheap judges but still below the frontier.

That gap is the main caveat. OS-Shepherd-9B trails Claude-Opus-4-8 by about 9.5 percentage points on the hard set, so the authors do not claim a fully reliable low-cost judge. Fine-grained scoring remains weaker still: the strongest judge’s quality rating reached only 63.5%, far below its binary accuracy. The paper’s contribution is a shared yardstick and an open starting point. It shows that reward signals can be standardized and made cheaper, but determining whether an agent truly changed the environment—not just claimed to—remains unsolved.