AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Android Agents Can’t Roll with the Punches: New Benchmark Exposes Runtime Blind Spots

Mobile AI agents that operate Android phones by reading screens and tapping buttons can handle scripted tasks—but stumble badly when the real world interrupts. A new benchmark called ANTRAP, from researchers at Zhejiang University, Yale, Tongji, and the University of Chinese Academy of Sciences, systematically injects runtime anomalies—unexpected pop-ups, mis-taps, freezes, and other disruptions—into agent trajectories to test whether these systems can recover. The results, posted as a preprint, suggest current GUI agents are fragile: every one of 16 leading models from seven organizations performed worse under injected disturbances.

The researchers built ANTRAP on AndroidWorld, a dynamic emulator environment, and expanded its task set from 116 to 236 tasks. They created a taxonomy of ten trap types across four layers: State (pop-ups, visual obscuration), Thinking (temporal conflicts, hallucinations), Action (grounding errors, wrong action types, intent deviation), and Round (deadlock, context disruption, loops). Perturbations are injected randomly during execution, and tasks are designed to remain solvable, so failures can be attributed to inability to recover rather than impossible instructions. The study reports rule-based Pass@3 success rates, giving credit if any of three attempts succeeds. Human annotators scored 93.4% on trapped tasks, confirming the tasks were not inherently unsolvable.

In one illustrative failure, the open-source model GUI-Owl-7B was running the “ExpenseDeleteMultiple” task when a grounding-error trap shifted its click coordinates by 20–50 pixels before execution. The model continued as though the click had landed correctly and ultimately failed the task. The authors note that such traps probe whether an agent can detect mismatches between what it intended and what actually happened on screen.

Across the full benchmark, Claude-Sonnet-4.6 fell from 74.2% success on clean tasks to 66.5% on trapped tasks, a 7.7 percentage-point drop; GUI-Owl-1.5-32B-Think fell from 69.5% to 62.4%. Thinking-style models did not make agents more robust in these tests: Qwen3-VL-8B-Thinking dropped 6.1 points (62.7% to 56.6%), roughly matching the 5.8-point drop of its non-thinking Instruct counterpart (52.1% to 46.3%).

The researchers also trained two models—GUI-Owl-7B and UI-TARS-1.5-7B—with Group Relative Policy Optimization (GRPO), a reinforcement learning method, in both clean and adversarial environments. Clean-environment training improved ordinary task performance (GUI-Owl-7B rose from 63.1% to 69.9%) but did little for robustness. Training on injected traps produced clear gains for single-step disturbances: state-layer traps improved by 8.1 to 11.0 percentage points, and action-layer traps by up to 8.5 points. But thinking-layer gains were smaller, and multi-step round-layer traps improved by less than 3.0 points; loop traps gained under 1.0 point. The authors attribute this to agents favoring current-screen information over long-horizon context, an explanation rather than a proven mechanism.

A key limitation: ANTRAP contains only 236 base tasks, which may not cover the full range of Android usage. The authors also describe the benchmark as diagnostic, not a general training recipe, and note that adversarial supervised fine-tuning remains unexplored. The work’s broader value is in naming a concrete problem—runtime robustness—and offering a way to measure it, even as contextual failures remain difficult to address.