AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Benchmark Finds World Models Falter When Interaction Tests Reliability

World models—AI systems that learn to predict how an environment will respond to actions—are often judged by how realistic their generated frames look. But a realistic image is not enough if the simulated world loses track of objects, ignores a command, or breaks its own rules after a few seconds. HappyWorld-Bench, a new evaluation suite from Zhiqi Bai and colleagues, argues that these systems should be tested on whether their worlds remain reliable under exploration, interaction, and modification.

The benchmark organizes world modeling into six hierarchical capabilities. W1, perceptual world, covers coherent scene construction. W2, interactive world, tests action-conditioned transitions. W3, persistent world, checks memory and identity over long rollouts and revisits. W4, programmable world, tests explicit interventions such as natural-language rule edits. W5 and W6 extend toward shared, scalable and universal worlds. The team operationalizes this taxonomy across three tracks: video world models, spatial world models, and embodied world models. It includes 1,138 video prompts, 300 spatial scenes, and 254 embodied test cases—1,692 track-specific instances—and evaluates 14 video models, 9 spatial systems, and 8 embodied candidates.

The method combines human and automated evaluation. HappyWorld-Arena runs A/B comparisons to produce model-level Elo ratings, a relative measure of human preference. Alongside that, automated metrics score behavioral correctness: instruction following, state consistency, physical causality, trajectory accuracy, edit success, and preservation of non-target content. Spatial tests use rendered views and exported geometry; embodied tests condition each candidate on a single egocentric frame and an action prompt.

A concrete W4 example shows the standard. After a gravity intervention, the paper says later motion should follow the modified rule—showing a low-gravity response and a stable landing when gravity returns—while unrelated scene content, such as the floor and curtains, retains its appearance and state.

The results reveal gaps. In the video track, HappyOyster received the highest Arena Elo, 1263, ahead of Genie 3 at 1206. At the capability level, Genie 3 scored highest on W1 with 82.8, slightly above HappyOyster’s 82.4, while HappyOyster led the reported W2 and W3 scores with 78.0 and 76.8. For W4, only HappyOyster and Lingbot-World-v2 were compared, scoring 66.7 and 59.3. The authors report that video models lose consistency during extended rollouts and revisits. In the spatial track, systems reached at best 70.14% placement accuracy and 73.33% edit execution. Embodied models struggled to preserve state across multi-step actions and to respond precisely to altered action conditions and physical rules.

The authors caution that the W4 comparison covers only two systems, and W6 is unscored. They also note that the spatial track uses static scenes because no evaluated method achieved spatiotemporal world modeling, and that W2 and W3 use different case pools, so their score differences are not a pure duration effect. The broader implication is that world-model progress should be measured not only by visual quality but by state consistency and correct responses to actions and interventions. The benchmark does not prove that current models are unusable, but it shows that reliability under interaction remains an open problem.