AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Computer-Use Agents Often Miss What the Person Asked For

Computer-use agents, AI systems that operate web browsers and desktop software on a user’s behalf, are typically graded against fixed task descriptions. Real people, however, leave things unsaid, add personal constraints, and sometimes want to approve an action before it happens. A new study argues that completing a fixed task and serving the person who asked are different goals, and that current benchmarks cannot tell them apart.

The authors introduce PersonaCUABench, which holds 74 web tasks across 11 domains constant and varies only who makes the request. Tasks come from two existing benchmarks: information-gathering on live websites and action tasks on sandboxed replicas of real sites. A “persona agent,” a language model playing a user defined by a 37-attribute profile, rewrites each instruction in its own words. The computer-use agent sees only this dialogue, never the profile or the original instruction. It may ask clarifying questions, which the persona answers in character. The personas fall into four groups: expert, senior, young, and practical.

The researchers tested nine backbone models, the underlying AI systems that drive the agents, over 3,330 runs. Averaged across models, task completion fell from 0.689 on original instructions to 0.441 on persona requests. That is a relative drop of 36 percent, or 24.8 percentage points on a 0-to-1 scale. Each model’s relative loss ranged from 28 to 45 percent. Senior personas scored lowest, at 0.402, and expert personas highest, at 0.504, still below the original.

Satisfaction showed a similar gap. Persona agents rated delivered results on a 1-to-5 scale, averaging 4.04 for original requests and between 3.10 and 3.59 across persona groups. Among persona runs that were fully completed, only 62.7 percent received the top rating of 5, compared with 84.1 percent of fully completed original requests.

To explain the drop, the authors assigned each of 1,230 losing runs a primary cause. In 59.9 percent, the agent engaged with a requirement the user had raised but still got it wrong. In 11.2 percent, the agent stopped where the user had asked it to, such as drafting something for review, and the automated checker scored that stop as a failure.

One example illustrates the first failure mode. A practical user asked for “a simple carnivore-style dinner option from the BOGOs,” meaning buy-one-get-one offers. DeepSeek V4.1 Flash returned several meat options rather than one dinner pairing. It met three of four rubric items, and the user rated the answer 2 out of 5, objecting that it was “a list of options rather than one specific pairing.”

Two interventions on GPT-6 Astra did not close the gap. Showing the agent the user’s profile did not recover task completion, and requiring at least one clarifying question moved persona completion only from 0.434 to 0.421. Only 24 percent of the 609 questions asked in the main experiment targeted a requirement the user had left out.

The study has clear limits. Personas and satisfaction ratings come from language models, which the authors acknowledge may encode stereotypes and cannot stand in for real users. Each episode is a single request on web tasks, not a long-term relationship with an assistant. The authors argue that agents should be judged on the requirements users actually express, not only on fixed specifications. The results, drawn from one benchmark and simulated users, identify a measurement problem worth testing with real people, but they do not establish how large the gap is in everyday use.