AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Teaching Vision-Language Models to Imagine Unseen Views, With Uneven Gains

Vision-language models (VLMs), which answer questions about images, often struggle when a question requires reasoning about what an image does not show, such as how a scene looks from another angle or how a stack of blocks would fall. A research team from The Hong Kong Polytechnic University and Hangzhou Diagens Biotechnology describes a method for addressing this gap. Their preprint, posted to arXiv, presents CoWorld Reasoner, which lets a VLM request imagined visual evidence from a video world model (VWM), a generative system that produces video of plausible scene changes.

Earlier approaches typically connect a video model to a fixed pipeline, most often to synthesize new viewpoints. The authors argue that different questions need different kinds of imagination. Their framework therefore exposes four operations as tools: view change, interpolation between observed states, continuation of observed motion, and intervention on the scene. Written “skills” guide the VLM in deciding whether to imagine, which tool to call, how to phrase the request, and how to interpret the resulting video. A verifier describes events in the generated clips.

The paper’s illustrations show the logic. Asked, from a robot’s point of view, whether a red cone is left or right of a blue cube, the system can request a view change, generate footage from a connecting perspective, and compare it with the original images. The authors’ figure shows the model concluding that the red cone is right of the blue cube.

The system is refined through what the authors call VLM-VWM Harness Co-Evolution. Across four stages, a supervisor running the same VLM reviews execution traces and revises tools, skills, and code. Neither model’s weights change. Refinement used 120 development examples held out from testing.

Using GPT-5.6 as the agent, supervisor, and answer scorer, and MiniMax H3 Max Turbo as the video model, the authors tested seven benchmarks covering spatial, temporal, causal, and perceptual reasoning. Average accuracy rose from 58.29% under direct GPT-5.6 reasoning to 59.69%, a gain of 1.40 percentage points. The largest relative gains were on MMSI, where accuracy rose from 38.35% to 42.89%, or 4.54 percentage points (about 11.8% relative), and on ERQA, which rose from 51.75% to 58.50%, or 6.75 points (about 13.0% relative). The system improved on all seven benchmarks and outperformed five evolving-agent baselines on five of them. The strongest baseline, Meta-Harness, averaged 57.63%.

The gains were uneven. On CausalSpatial, the system improved by only 0.07 points. The authors report that video tools contributed most on occlusion and collision subtasks, adding 5.36 and 2.55 points respectively, which they attribute to generated video revealing intermediate interactions. Substituting the weaker LTX-2.3 video model produced smaller gains and left CausalSpatial below the GPT-5.6 baseline.

Several caveats apply. The approach depends on the quality of generated video, and the authors do not report the computational cost of repeated generation and co-evolution rounds. The development set is small, and all experiments rely on one model family. When transferred to Grok-4.5 with one additional co-evolution round, overall accuracy rose from 32.86% to 39.78%, though SAT declined slightly.

The work suggests that a VLM can learn which kind of visual imagination a question calls for, rather than following one fixed pipeline. Whether such gains generalize beyond these benchmarks remains an open question.