AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Beyond the Textbox: New Benchmark Reveals AI Still Struggles to Learn from Visual Context

Large language models are celebrated for “in-context learning”—the ability to process novel instructions and data provided on the fly without needing to be retrained. However, while AI handles textual context remarkably well, real-world information rarely lives in pure text alone. Financial performance is tucked inside dense corporate reports, scientific breakthroughs rely on intricate figures, and urban navigation requires deciphering complex transit maps.

To evaluate how effectively modern AI learns from multimodal inputs, researchers from Shanghai Jiao Tong University and partner institutions have introduced CLBench-V, a benchmark comprising 3,443 diverse challenges. Their findings expose a stark reality: even the most advanced vision-language models struggle to reliably process, apply, and learn from visual context, with the top-performing system scoring just 28.5% overall.

To pinpoint exactly where AI systems break down, CLBench-V organizes tasks into a three-level hierarchy:

  1. Context Grounding (L0): Tests whether a model can simply locate and read visual details. For instance, given a street photo, can the model spot a small parking sign reading “Max. 14 days” to extract the number 14, or reliably trace a route across a complex subway map?
  2. New Information Application (L1): Tests whether an AI can feed freshly provided visual facts into procedures it already knows. For example, can it scour multi-page financial reports, correctly match revenue and equity figures across tables, and calculate a company’s Return on Equity (ROE) without accidentally swapping values between adjacent columns?
  3. New Knowledge Learning (L2): The hardest test, evaluating whether a model can infer brand-new rules or scientific conclusions directly from visual context. For instance, when presented with the methodology and result graphs of an unfamiliar medical paper, can the model synthesize the underlying empirical findings without relying on pre-existing training memory?

Across evaluations of six major model families—including InternVL3.5-30B-A3B, Qwen3.5-Plus, Kimi-K2.6, and GPT-5.4—no single system dominated all capabilities. InternVL achieved the highest overall score (28.47%) and led in grounding visual details (L0) and synthesizing novel scientific conclusions (L2). Meanwhile, Qwen3.5-Plus excelled at applying document-specific facts and chart values (L1).

The benchmark highlighted three recurring failure modes:

  • Evidence Misbinding: An AI locates the right visual visual element but attaches it to the wrong label—such as reading a survey chart correctly but assigning responses from “Democrat” voters to a “Neither Party” category.
  • Prior Override: The model ignores newly provided visual evidence in favor of outdated facts memorized during pre-training.
  • Incomplete Induction: The AI notices isolated visual details but fails to combine them to discover an overarching rule or trend.

The study demonstrates that simply expanding an AI’s ability to accept multiple images or long visual inputs is not enough. As AI deployments expand into web browsing, financial analysis, and scientific discovery, models must move beyond merely “seeing” visual evidence to truly learning from it.