AI Papers Reader

Personalized digests of latest AI research

View on GitHub

The Illusion of the AI Scientist: Why Today’s LLMs Still Falter at Real Scientific Discovery

Large language models (LLMs) are rapidly stepping into academic labs, writing code to analyze complex datasets and automated research pipelines. But a critical question remains: Is an AI that can write flawless code actually doing sound science?

According to a new study from researchers in China and Hong Kong, the answer is a resounding “not yet.” In a paper introducing a new evaluation framework called SDABench, researchers argue that existing benchmarks conflate computational success with scientific validity. Just because an AI writes Python code that executes without error does not mean it has drawn a scientifically sound conclusion.

To expose this gap, SDABench moves beyond code execution. Instead, it evaluates LLMs on six progressive scientific discovery capabilities—descriptive, exploratory, inferential, predictive, causal, and mechanistic reasoning—across five core disciplines, including biology, chemistry, and physics. The benchmark utilizes a mix of 527 real-world scientific datasets (SDA-Real) and 6,000 synthetic datasets (SDA-Synth) to test the models.

To understand why this distinction matters, imagine an AI analyzing how plants respond to drought. If asked a simple descriptive question—such as calculating the average soil moisture in a dataset—modern LLMs excel. But science rarely stops at averages.

If the task shifts to exploratory analysis, the AI might mistake random statistical noise for a groundbreaking correlation between a rare soil nutrient and leaf size (a “Relationship Error”). If pushed to inferential or causal tasks—such as proving whether a specific genetic tweak actually causes drought resistance or is simply a bystander effect—the AI must understand latent processes and statistical boundaries. Here, the researchers found, even the most advanced models degrade sharply. They can run the mathematical calculations, but they struggle to select the correct statistical assumptions or model the hidden physical mechanisms.

The researchers tested 15 representative LLMs using both multiple-choice and open-ended questions. Interestingly, they found that multiple-choice tests often mask an LLM’s true weaknesses. When presented with pre-written options, models frequently rely on superficial pattern matching to guess the right answer. When forced to answer open-ended questions where they must explain their reasoning and construct the answer from scratch, their performance plummeted.

Furthermore, the study introduces a five-stage error pipeline to trace precisely where the AI’s reasoning breaks down: Scope, Variable, Function, Relationship, and Conclusion.

By classifying these errors, the researchers discovered a telling trend. As LLMs become larger and more sophisticated, they get much better at the basics. They rarely make “Variable Errors” (such as grabbing the wrong column of data) or “Scope Errors” (misunderstanding the goal of the experiment). However, they still hit a wall at the later stages of reasoning. They frequently suffer from “Function Errors”—choosing the wrong statistical formula for the data structure—and “Relationship Errors,” where they fail to map out how variables interact.

Ultimately, the paper suggests that building an autonomous AI scientist requires more than teaching LLMs to write clean code. Future AI development must focus on teaching these models how to evaluate scientific assumptions under uncertainty and verify their logic. Until then, AI may remain an excellent lab assistant, but the crown of the true scientist still belongs to humans.