AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Search Agents Learn Better When Retrieval Becomes a Menu, Not a Query Box

Reinforcement-learning (RL) search agents typically retrieve information by having a language model type free-form natural-language queries, then optimizing the whole multi-turn interaction with a single final-answer reward. Researchers from the National University of Defense Technology argue that this interface itself undermines training. In experiments with Search-R1, they observed what they call retrieval-equivalence collapse: for the same question, the model kept generating different query strings, but the evidence the queries pulled back became increasingly identical. That left rollout groups with little genuine retrieval contrast, so group-relative RL had almost no useful signal about which retrieval actions were better.

Their proposed framework, Harness-G, changes the action space rather than the reward scheme. The corpus is organized offline into a graph with paragraph, sentence, and entity nodes. At each step the environment builds a finite action menu: Select commits a visible sentence as evidence, Lookup retrieves around a named entity, and Answer stops and generates the final answer. The policy never writes a query; the environment constructs and executes it. The menu is deduplicated, filtered, and previewable, so alternatives at the same decision point are explicit and comparable. A second component, Structured Non-myopic Credit (SNC), uses a frozen answer scorer to compare the chosen action to same-state alternatives and propagates later answer gains back to earlier actions that enabled them.

A held-out 2WikiMultiHopQA case illustrates the mechanism. Asked for the birth date of the director of The Metamorphosis of Mr. Samsa, a free-query agent rewrote its searches but eventually answered with the birth date of distractor director Satyajit Ray. Harness-G with Qwen2.5-3B instead selected the sentence linking the film to Caroline Leaf, looked up Leaf, committed a sentence giving her birth date, and answered August 12, 1946. The typed actions kept the bridge hop and the attribute hop distinct.

Across six QA benchmarks, Harness-G achieved the highest average F1—token overlap between generated and gold answers—at both tested model scales. With Qwen2.5-3B, average F1 was 55.24 versus 51.26 for Graph-R1, a gain of 3.98 points; with Qwen2.5-1.5B, it was 50.83 versus 40.09, a gain of 10.74 points. The largest gains were on multi-hop datasets, and the method trailed Graph-R1 on NQ by 1.84 F1. The authors’ ablations indicate both the menu and SNC contribute. Training cost was modest: programmatic graph construction incurred no API cost, and SNC added 9–11% per-step wall-clock time without extra rollouts.

The system is text-only, however, and the appendix shows cases where the gold answer was not on the menu or the final answer string still mismatched committed evidence. The authors describe Harness-G as evidence that action-space design is a complementary axis to reward design for training search agents—not as proof that free-form search is obsolete.