AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Fluent AI Narrators Frequently Contradict Their Own Stories, Stress-Test Finds

Large language models are increasingly cast as game masters and interactive storytellers, but keeping a story world coherent over many turns is hard: a narrator may establish that a door is locked, then let a character walk through it. A new benchmark, NCP-Bench, formalizes this problem as Narrative Commitment Preservation and measures it across 100 environments derived from movie synopses.

The central idea is to make consistency checkable. Each environment starts with an initial fact ledger of established world state, a set of plot commitments, and an ordered reference trajectory of major plot steps. A narrator agent responds to free-form player actions, and a fixed auditing pipeline checks the response against established facts, commitments, and the player’s stated intent. A confirmed conflict ends the episode; success requires all achievement commitments to be satisfied without any violation. The player is an adversarial agent that tries to expose weaknesses by skipping plot steps, making premature accusations, or probing contradictions, while using only information the character could know.

The failures are concrete. In an environment based on Alien, the fact ledger recorded that Dallas was on the bridge and had ordered Ripley to report her location by intercom. The narrator later wrote: “But standing in the doorway… is Dallas. He’s not on the bridge. He’s here, waiting”—with no narrated movement between locations.

Across six state-of-the-art LLMs, fluent text was no guarantee of logical stability. GPT-5.2, the strongest model by survival, still had only a 42% survival rate after 20 turns. Fact conflicts were the dominant failure, appearing in 40% of interactions for GPT-5.2 and up to 68% for Qwen3-235B-A22B. Only isolated runs satisfied all achievement commitments within the 100-turn limit; just 3.5% of all runs reached 100 turns without an explicit conflict, and most of those did not complete the required goals. The authors interpret this as a behavioral split: DeepSeek-V3.2 reached the highest trajectory progress (15.40%) and satisfied-commitment rate (13.42%) despite averaging only 15.88 turns, while GPT-5.2 lasted longer on average (32.92 turns) but reached only 9.94% trajectory progress.

An explicit memory architecture changed the failure profile but did not solve the task. HiAgent, a memory-augmented version of GPT-4o-mini, increased average turns from 22.16 to 30.05 and cut commitment conflicts from 26% to 4%, but player-input conflicts rose from 13% to 38%, and no run satisfied all achievement commitments.

The authors caution that NCP-Bench targets linear, single-threaded narratives; branching timelines, flashbacks, and parallel perspectives remain untested. Auditor LLMs can also misjudge ambiguous text, though a human review of 100 GPT-4o-mini runs disputed only four fact-conflict flags. They also note that logical consistency alone does not make a compelling story; dramatic tension and emotional resonance are left to future benchmarks.

Still, the researchers see a wider lesson. The same requirement appears wherever an AI system must honor prior commitments under open-ended input: coding assistants that must not silently break existing behavior, multi-turn dialogue systems, tool-using agents, and multi-agent coordination. NCP-Bench, in their view, is a controlled testbed for that broader problem—not proof that a solution is near.