AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Show Your Work: New AI Benchmark Demands Pixel-Perfect Proof in Video Questioning

Imagine asking an AI model watching a home video, “Why did the man fall down?” The AI might instantly reply, “Because the ladder slipped.” It sounds impressive, but does the AI actually see the ladder, or is it just guessing based on common sense? Today’s cutting-edge Video Large Language Models (Video LLMs) are notorious “black boxes.” They excel at generating correct textual answers but offer no way to prove they actually perceived the visual cues, raising critical safety concerns in high-stakes fields like autonomous driving, medical diagnostics, or robotic surgery.

To crack open this black box, researchers from Salesforce and Brown University have introduced Evidence-Backed Video Question Answering (E-VQA). Rather than letting AI off the hook with a simple text response, E-VQA demands a “triplet” of proof: the textual answer, the exact video segment where the event occurs (temporal evidence), and precise, frame-by-frame highlighted outlines of the objects involved (spatial evidence, known as “masklets”).

Let’s build an intuition for how this works. In a video of a backyard accident, a standard AI might correctly answer that a person fell because “the ladder slipped backward.” However, under the hood, the model might actually be focusing on a static green wall in the background, or simply relying on language shortcuts. Under the new E-VQA standard, the AI cannot cheat. It must explicitly pinpoint the exact video timestamps—say, from 8.6 to 11.0 seconds—and physically paint a digital mask tracking the slipping ladder and the falling climber frame-by-frame. If it fails to track the ladder, its “correct” answer is flagged as ungrounded.

To test this, the researchers built ST-Evidence, a first-of-its-kind, human-verified benchmark, and put top-tier models like OpenAI’s o3 and Google’s Gemini 2.5 Pro to the test. The results were startling: high answering accuracy does not equal true visual perception. Even when models got the text answer right, their ability to pinpoint the physical evidence was often no better than random guessing. Simply making the models larger (scaling) failed to close this gap, proving that larger brains do not guarantee better eyes.

To solve this, the team developed ST-Evidence-Instruct, a massive training dataset of 160,000 video-reasoning triplets. Because manually drawing frame-by-frame outlines for thousands of videos is incredibly laborious, they designed automated, bidirectional generation pipelines. These pipelines either take existing video masks and generate logical questions about them, or take complex QA pairs and decompose them to generate precise bounding boxes and track them using advanced tracking models.

Fine-tuning specialized models on this dataset yielded massive gains. A 7-billion-parameter model trained on ST-Evidence-Instruct saw its temporal evidence accuracy jump by 27.2 percent and its visual tracking accuracy boost by a staggering 13.8 percent.

By forcing AI to “show its work,” E-VQA marks a crucial shift from blind trust to verifiable reasoning, bringing us one step closer to reliable, transparent machine vision.