AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Even Top AI Agents Struggle With Multi-Step Research on the Live Web

AI agents that search the web, click through pages, and read documents are increasingly marketed for research, but their ability to complete long, multi-source investigations on the open internet has been hard to measure. A team of researchers from several universities, led by the Hong Kong University of Science and Technology (Guangzhou), reports that the best of 12 frontier models answered fewer than two in five of the questions in a new benchmark. Human solvers averaged 21.1 percent, below the top model but above five of the twelve models tested.

The researchers define a task called OpenDS (Open-Web Deep Search). An agent receives a factual question and an initial clue, then must find, access, and process evidence across web sources, connecting them to reach one verifiable answer. Real web information is spread across text, images, audio, and video, and appears in formats such as HTML, PDF, and Parquet, a structured data format. Some evidence is also locked behind interactions such as navigation, parameter entry, or file downloads. The benchmark, called OpenDS-World, contains 200 tasks across eight domains, including culture, science, history, and economy.

Each task begins with a public video and an entity that can be identified only from its audio and visual content. Expert annotators then extend an evidence path step by step, using Google Search to find sources that corroborate each new entity. Once the path reaches a unique endpoint, the annotators rewrite it as a set of indirect clues, so that no single clue reveals the answer. Ten reviewers checked the evidence and the uniqueness of each answer, and tasks that failed were discarded.

The paper’s illustrative task shows the chain. It starts with a museum presenter’s video, requires locating a fossil display in an official virtual tour, follows a podcast episode and a cited paper’s chart, moves to a 2018 park video, and ends with a lookup in a UNESCO Parquet dataset. The correct answer is ASTH01, the identifier for Satun UNESCO Global Geopark in Thailand. Each hop requires a different medium or access method, so an error at any stage derails the result.

On the benchmark, GPT-6 Astra scored 39.0 percent, followed by Claude Opus 5.5 at 38.0 percent and Gemini 3.8 Flash at 36.5 percent. Qwen3.8 Flash scored lowest, at 12.0 percent. The human average of 21.1 percent came from 1,000 independent attempts, and 42.5 percent of tasks went unsolved by every human assigned to them. The top model’s score is 17.9 percentage points above the human average, though this is not a relative gain of 17.9 percent. The authors observe that more tool calls and longer runs did not reliably produce higher accuracy. They attribute this to weaknesses in choosing sources, coordinating tools, adjusting strategy, and managing long searches, though these explanations are their interpretation rather than something the experiments isolate.

The comparison with humans has clear limits. Human solvers could use standard browsers but not large language model tools, and no time limit applied, so the baseline is not a like-for-like test of agents. Each of the ten question sets was attempted by only five solvers, so the human estimate rests on a modest sample. The benchmark also depends on one unified tool environment, and it is unclear how closely results would carry over to other setups.

The authors position OpenDS-World as a shared platform for testing cross-modal, cross-format, and interactive research. The results suggest that current agents remain well short of dependable open-web investigation, but they do not establish how far the benchmark’s difficulty reflects real-world research demands in general.