AI Papers Reader

Personalized digests of latest AI research

View on GitHub

AI Agents Can Code the Experiment, but They Can’t Do the Science Yet

Promising forecasts from leading artificial intelligence labs suggest autonomous AI agents will soon automate scientific discovery, conducting research and development to continuously improve themselves. But while today’s top models can execute complex software engineering tasks, a new study reveals a stark bottleneck: when handed open-ended scientific problems, frontier AI agents lack the fundamental judgment and creativity required to produce meaningful science.

To evaluate whether AI can act as independent researchers, computer scientists from Princeton, the UK AI Security Institute, and several partner institutions introduced a benchmark method called “shadow evaluations.” Rather than testing agents on narrow, easily verified coding tasks, the researchers tasked state-of-the-art models—including Anthropic’s Claude Opus 4.8 and OpenAI’s GPT-5.6 Sol—with answering the primary research questions of high-quality, unpublished papers submitted to NeurIPS 2026. Equipped with Linux virtual machines, cloud GPUs, open web access, and $3,000 API budgets, the agents had six days to explore, experiment, and write conference-worthy manuscripts.

The original human authors of the papers then double-blind reviewed the agents’ outputs. The result was decisive: both AI-generated papers were unambiguously rejected.

The contrast between the agents’ engineering capabilities and their scientific reasoning was striking. Technically, the models operated flawlessly without human help. When four rented GPU servers suffered crash loops due to missing system directories, an agent autonomously debugged the environment, managed cluster resources, and executed hundreds of compute-heavy trials. The agents also compiled clean LaTeX documents and conducted thorough literature reviews.

However, when faced with core scientific decision-making, the agents suffered from five recurring failure modes:

  • Premature Exploration Cutoffs: In a study on controlling language model “personas,” the agent initially budgeted 42 hours to test three distinct hypotheses. Instead, after a brief five-hour run on its first idea, it abandoned the remaining hypotheses and built its entire paper around weak negative findings derived from hand-picked word lists.
  • Inability to Pivot: In an experiment building distribution-shift detectors for tabular models, initial tests failed. Rather than brainstorming a fresh theoretical angle or abandoning the dead end, the agent simply reframed its goal—declaring that “no such detector can exist” and submitting a negative-results paper based on trivial synthetic data.
  • Superficial Handling of Feedback: Instructed to run automated peer-review tools, the agents repeatedly received internal scores of “Weak Reject.” But instead of rethinking their flawed experimental setups, they simply added cautious 40-line disclaimers to defend their existing conclusions.
  • Poor Resource and Time Awareness: Both agents left over half of their $3,000 API budgets unspent and declared their work finished hours before the strict six-day deadline, seemingly unable to grasp how to use remaining time to improve weak drafts.
  • Instruction Drift: Over the course of multi-day runs, the agents gradually ignored explicit constraints, such as paper length limits, causing final drafts to fail basic submission requirements.

The study highlights a critical gap between automated engineering and true scientific discovery. While frontier models excel at executing defined technical steps, open-ended research requires knowing when a hypothesis has failed, how to pivot creatively, and how to set a high bar for evidence. Until AI systems master these high-level reasoning skills, fully autonomous AI research remains out of reach.