AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Mind the Gap: New Benchmark Exposes Embarrassing AI ‘Blind Spots’ on Simple Tasks

Today’s state-of-the-art artificial intelligence models can ace bar exams, write complex computer code, and diagnose rare medical conditions. Yet, if you ask those same frontier systems to generate an image of a dog with exactly five legs, or to write a sentence that has exactly 31 characters, they often fail spectacularly.

To bridge this confusing divide, researchers from École Polytechnique Fédérale de Lausanne (EPFL) in Switzerland have introduced blind-spots-bench. This new benchmark of 235 curated questions is specifically designed to expose the persistent “blind spots” where advanced AI falters on tasks that humans find nearly trivial.

Trivial for Humans, Hard for AI

To understand these blind spots, consider a common word-search puzzle. A human child can scan a grid of letters and find the word “SEARCH” with ease. However, advanced vision-language models (VLMs) frequently misidentify or completely overlook letters in cluttered visual environments—a failure of “attribute and pattern recognition.”

The EPFL researchers collected these frustratingly simple failures by asking graduate-level AI students to find prompts that tripped up the world’s best chatbots. They classified these challenges into three broad categories:

  • Object-centric tasks: Such as counting the number of airplanes in a photo or generating an image of an upside-down tree under a building.
  • Abstract reasoning: Such as solving a basic Sudoku puzzle or finding the shortest route on a simple map.
  • Language-and-knowledge tasks: Such as manipulating text at the character level (for example, repeating a number sequence exactly 31 times without spaces).

The Closed-Source Edge and Scaling Surprises

Evaluating 38 state-of-the-art models—including OpenAI’s GPT series, Google’s Gemini family, and open-weight models like DeepSeek and Qwen—unveiled several eye-opening insights.

First, proprietary, closed-source models still hold a significant edge. Frontiers like Gemini-3.1-Pro and GPT-5.5 outperformed their open-weight counterparts by roughly 10 percent on the benchmark. Surprisingly, this gap exists even when the open-weight models boast comparable scores on standard academic tests. This suggests that traditional public benchmarks may be overestimating how robust these open systems actually are on underrepresented tasks.

Second, simply making a model larger does not guarantee it will get smarter. In the Qwen3.5 family, for instance, a massive 122-billion-parameter model actually suffered a 10 to 14 percent performance drop on several abstract reasoning tasks compared to its much smaller 35-billion-parameter sibling.

The Visual Counting Bottleneck

The benchmark also highlighted areas where all AI models are collectively struggling. Fine-grained visual perception remains a major hurdle. Even the most powerful visual-language models topped out at less than 60 percent accuracy when tasked with basic perceptual counting, such as counting chess pieces on a board.

Even giving models access to tools, like letting them run Python code to double-check their math, yielded mixed results. While some models improved, others actually performed worse because they botched simple steps like copy-pasting their inputs into the code.

Ultimately, the researchers argue that blind-spots-bench serves as a crucial diagnostic stress test. By shifting the evaluation focus away from hyper-complex exams toward the basic cognitive skills humans take for granted, this benchmark provides a realistic roadmap for building truly reliable AI.