AI Papers Reader

Personalized digests of latest AI research

View on GitHub

World Action Agent Lets Vision-Language Models Pilot Robots Through Rehearsal

General-purpose vision-language models (VLMs) can reason about images and language, but turning that knowledge into precise robot control remains difficult. Existing systems often use them to predict constraints or write programs, or show them a scene rather than a world for testing actions. A new paper proposes World Action Agent (WAA), a multi-agent harness that lets a VLM pilot a robot using basic tools inside a visual action workspace.

WAA changes what the VLM sees and how its decisions take effect, rather than retraining the model. The harness selects Contact views—close-ups centered on the current interaction, plus global context—so local relations among gripper, object, and target are visible. Each action begins as an editable proposal. The harness solves inverse kinematics, plans with cuRobo, and reports feasibility. An Imagination Agent can rotate or shift a pose while the physical scene stays unchanged. Only an executable proposal becomes motion. After execution, the agent corrects residual offsets in the view where it observes them.

In one recorded bowl-placement task, Contact A appeared aligned, but Contact B revealed an offset. The main agent dragged the bowl toward its target in Contact B, and the harness converted that image-space drag into bounded end-effector motion. After correction, the bowls were stacked. The authors say such corrections address depth noise, calibration error, and contact disturbances left by planned motions.

The researchers also build procedural knowledge through the same interface. Skills start as text-only seeds, then evolve from one expert video per LIBERO-90 task and from human teaching. A Learner extracts evidence, an Editor revises skills, and a Reviewer checks source frames and consistency. The library stays frozen during evaluation. Separately, interaction traces train smaller VLMs to operate the harness.

On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reached 75.6% average success, versus 72.0% for ASPIRE. It led on both Spatial splits (80.0% and 73.3%) and Goal Task (48.3%), while ASPIRE remained strongest on both Object splits and Goal Pos. Zero-shot WAA reached 28.9% average success, compared with 6.7% for Show-Harness with the same Gemini 3.7 Flash backbone. The frozen LIBERO skills transferred to robosuite without further learning, raising the three-task average to 100.0% from 86.7% without skills. Fine-tuning Qwen3.5-9B on harness traces raised out-of-domain success from 1.7% to 43.3%, and in-domain success from 0.0% to 55.0%.

The authors note several limits. WAA’s performance remains bounded by its backbone; Gemini 3.7 Flash still integrates multi-view information imperfectly on some tasks, where WAA is less stable. The problem lies in the backbone’s perception rather than the harness interface, they write. Cost and speed also remain concerns: WAA averaged 31 model calls and 150 seconds per episode, versus 120 calls and 874 seconds for Show-Harness.

Still, the work suggests a different route: instead of fine-tuning VLMs into action predictors and risking their general reasoning, design an interface that lets them observe, rehearse, and correct actions. Whether it scales to messier real-world tasks and different VLMs remains open.