AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Qwen-UI-Agent Brings GUI Assistants From Clean Benchmarks to Real Phones

Graphical user interfaces are how most people use software, and “GUI agents”—AI programs that perceive screens and carry out clicks, taps, and typed commands—could become everyday digital assistants. So far, though, many agents are tuned on simulated benchmarks and struggle with real-device conditions: pop-ups, permission requests, unstable networks, and apps that change. In a technical report, Alibaba’s MAI-UI team introduces Qwen-UI-Agent, a multimodal foundation model built to close that gap across mobile, desktop, web, and search environments.

The system’s central design choice is a unified action space that lets the model combine GUI operations with command-line interface (CLI) commands and batch several actions into one decision step. GUI actions provide universal access to visual interfaces; CLI commands handle files, scripts, and system settings directly. The team also built a real-device mobile runtime with more than 100 physical phones and over 150 apps, plus sandbox environments. Training uses supervised fine-tuning and reinforcement learning, including online reinforcement learning on trajectories longer than 100 steps and up to 10,000 concurrent sandbox environments. An “AutoResearch-style” loop automates task creation, failure analysis, and next-round planning, with humans mainly supervising.

A paper example shows why this matters. When a flight-cancellation notification arrives on a user’s phone, the agent identifies affected plans, queries alternative flights through a search API, asks for approval before rebooking in a mobile app, updates a meeting schedule and spreadsheet on a desktop with GUI and CLI commands, and sends the revised files. The example illustrates cross-platform workflow and “proactive” service: the agent acts on a digital signal instead of waiting for an explicit instruction.

The authors report leading or competitive results. On MobileWorld-Real, a real-device benchmark with more than 400 tasks across over 100 apps, Qwen-UI-Agent achieved a 92.2% success rate, beating Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 6.0, 7.5, 6.8, and 3.5 percentage points, respectively. It scored 97.5% on AndroidDaily and 82.1% on the simulated MobileWorld, surpassing Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 14.6, 12.0, and 8.9 percentage points, respectively. On computer use, it reached 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2, topping MiniMax M3 by 17.7 percentage points and Qwen 3.7 Plus by 18.5 percentage points while using 58.4% and 21.7% fewer steps per task. It also scored 73.6% on WebArena and 81.5% on ScreenSpot-Pro. In computer-use tasks, CLI commands and GUI clicks were the dominant action types, and over 40% of action outputs were batched.

The authors stop short of claiming the gap is solved. High-risk actions still require human handoff for logins, payments, permissions, and CAPTCHAs, and the device scheduler must blacklist unreliable phones until manual repair. They also note executable verifiers are time-consuming to build, so they rely partly on model-based judges whose accuracy depends on model progress.

If these results hold, the broader significance is a step toward agents that can handle long-horizon, cross-platform tasks and proactively propose useful services. The report is a preprint; its value lies in emphasizing real-device training and evaluation over simulated perfection.