AI Papers Reader

Personalized digests of latest AI research

View on GitHub

A Ledger for AI Reasoning Makes Multimodal Agents Auditable

Multimodal AI agents — systems that pair a vision-language model with tools such as OCR, crop-zoom, and web search — increasingly answer questions through multi-step trajectories. Yet most evaluations still score only the final answer. That aggregate signal cannot show whether a correct reply came from grounded evidence or from a model’s prior guesses. Researchers at HKUST (Guangzhou) and collaborators introduce LedgerMind, a runtime designed to make the reasoning trail itself auditable.

LedgerMind treats an agent’s trajectory as a provenance-constrained state machine. Every tool output is normalized into an entry in a Structured Evidence Ledger, carrying source, type, confidence, lifecycle status, and dependencies. Reasoning and decision claims may cite only active entries. A three-layer grounding protocol then checks more than structural citation: it verifies that cited evidence actually contains the entities and numeric values used in the claim. An adaptive dispatcher routes simple queries to a direct path, avoiding expensive multi-step pipelines that can overwrite a correct answer. When violations are detected, repair is restricted to typed operators — refresh, retry, switch tool, abstain — so no new content can enter the ledger without tool-produced provenance.

The authors illustrate one failure mode they call phantom grounding with an Apollo 11 example: a model cites a document that says “Apollo 11, 1969” but produces the claim “Apollo 13, 1969.” The citation exists, but the cited evidence does not support the conclusion-level entities. LedgerMind’s entity consistency check is designed to catch such mismatches.

The empirical results span six public benchmarks plus an in-house stress test. On VTC-Bench, LedgerMind with Gemini-3-Flash scored 58.9%, higher than any proprietary tool-use baseline in the comparison. Compared with the same backbone run without the framework, it improved Gemini-3.1-Pro by 11.8 percentage points, Gemini-3-Flash by 12.4, and GPT-4o by 23.3. On EMMA, overall accuracy reached 58.29%, 9.58 percentage points above the strongest thinking-mode baseline; gains were largest in math (+16.15 pp) and physics (+16.02 pp), while coding fell by 0.18 pp. On the chain-level search benchmark MC-Search, LedgerMind with Claude-Opus-4.7 raised answer F1 from the best official baseline’s 41.78% to 61.28%, and recovery of golden intermediate steps from 31.35% to 57.82%.

Ablations on MMMU-Pro with Gemini-3-Flash show the ledger itself is the most important component. Removing it dropped overall accuracy by 15.39 percentage points, from 68.89% to 53.50%; replacing typed repair with free-form self-reflection cost 8.49 points.

The authors note important caveats. LedgerMind is evaluated only on static image-and-text benchmarks; extending it to video, long-horizon agents, or embodied interaction would require revisiting its time-to-live policy. The grounding checks use a fixed alias table and exact matching for years and counts, which may miss paraphrases or cross-lingual aliases. And because the framework is training-free, it inherits the biases and limits of the underlying models; its provenance guarantee means repair cannot invent unsourced content, not that tool outputs are factually correct.

The broader significance is in the audit trail itself. The authors suggest that ledger-style trajectories could help safety-sensitive uses such as tutoring, scientific diagram reading, and document understanding, and could eventually supply training signals that reward grounded reasoning. Those possibilities remain open; what the current experiments establish is that constraining provenance does more than add bookkeeping — across the benchmarks studied, it improved both accuracy and the faithfulness of the reasoning that produced it.