AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Simulated Users Put Visual AI Assistants to a Harder Test

Visual AI assistants that watch people perform physical tasks and offer guidance are being developed for cooking, housework, and phone settings. Most are judged by replaying recorded videos and comparing their replies with reference answers. That approach cannot show whether a piece of advice changes what a person does next. Researchers from the University of Washington and Salesforce AI Research describe a framework called Interactive Visual Gym that tests assistants in a closed loop instead.

In the framework, an assistant interacts with a simulated user inside an executable environment, and the user’s later actions depend on the assistant’s replies. The team built three simulators: CookSim, a 3D kitchen inspired by the game Overcooked; VHSim, an apartment environment adapted from VirtualHome; and ScreenSim, a replica of the iPhone 17 Settings app. A vision-language model plays the user, following a persona that varies in skill, trust in AI, execution speed, and interruption preference. The harness can also insert scheduled mistakes, such as skipping a step or adding a wrong ingredient, along with questions and plan changes. Because each environment tracks world state, researchers can check whether a task was actually completed. The framework also models latency: replies arrive only after a delay tied to measured inference time, and the simulated user keeps acting in the meantime.

The authors’ example shows how delay can undermine guidance. In a VHSim trial with a 12-tick delay, Gemini 3.1 Pro saw the user in a bedroom and told them to turn right toward the kitchen. By the time the reply arrived, the user was already in the kitchen, and the assistant’s instructions were out of date.

The benchmark, IVG-Bench, includes 315 scenarios: 150 in CookSim, 75 in VHSim, and 90 in ScreenSim. The best-performing model, Claude Opus 5.5, completed 53.5% of tasks within the time limit, averaged across the three environments. The authors report weaknesses in factual grounding and actionable guidance. Latency had a large effect on frontier models. In CookSim, Claude Opus 5.5’s in-time success fell from 57.1% under fixed two-tick timing to 31.8% with measured latency, a drop of 25.3 percentage points. Gemini 4 Argon fell 27.8 points, to 33.1%. The open-weight Muse Glimmer changed little, rising 2.0 points to 42.7%, and ranked first under measured latency. The authors attribute frontier-model losses largely to warnings that arrive after the user has already acted.

A human study with 45 participants ranking three anonymized assistants found that the interaction-quality metric agreed with participant preferences in 78.5% of pairwise comparisons. In a proof-of-concept reinforcement learning experiment, the team fine-tuned Qwen3.5-4B with GRPO, a training method that updates a model based on relative rewards across sampled attempts. With a Gemini 3.7 Flash user, in-time success on held-out validation scenarios rose from 33.3% to 42.2%, a gain of 8.9 percentage points after 12 updates. Detection recall, however, fell from 49.0% to 34.9%.

The study has clear caveats. Simulated users spoke to assistants far more often than human participants did, and the results depend on the language model playing the user. The training experiment used 30 validation scenarios and 12 updates, so it suggests a learning signal rather than demonstrated generalization across tasks or users.

The work offers a more realistic way to evaluate assistants that act during a task rather than after it. It does not show that any current model is ready to guide people in real physical settings, and the authors caution that performance in these simulators should not be taken as evidence of real-world safety.