AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Recursive Belief Scoring Gives AI Agents Better Credit for Pivotal Decisions

Training an AI agent to navigate a kitchen, shop online, or search the web usually ends with a single verdict: success or failure. That sparse outcome makes it hard for the model to learn which action in a long chain was decisive. Standard group-relative policy optimization (GRPO), a common method for fine-tuning language-model agents, assigns one sequence-level advantage to every token, treating all turns as equally responsible for the final result.

AgentOPSD, developed by researchers at Tsinghua University, Zhejiang University, and Meituan, is designed to fix that mismatch. During training, the same policy scores each generated token twice: once under the ordinary context and once with a “privileged” skill prompt describing useful subgoals. This is a form of self-distillation—teacher and student share parameters, but the teacher branch sees extra training-only information. The gap between the two log-probabilities acts as evidence about whether a turn supports eventual success. AgentOPSD sums the evidence over each turn, then uses it to update a Bayesian belief about the chance the trajectory will succeed, starting from the group’s observed success rate. Each turn receives credit equal to its marginal revision of that belief. The revision is aligned with the final verifier outcome and reshapes the GRPO advantage; no learned critic or extra rollouts are required, though the method adds one teacher forward pass.

To illustrate the failure mode, the authors describe two trajectories with identical final rewards, and therefore identical broadcast GRPO advantages: in one, success hinges on a single decisive turn; in the other, progress is spread evenly. A sequence-level advantage cannot distinguish them. AgentOPSD’s recursive belief revision can, because the same local gap is decisive while uncertainty is high but becomes redundant once accumulated evidence already points to an outcome.

The team evaluated AgentOPSD on ALFWorld (text-based embodied household tasks), WebShop, and Search-QA, using Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld with Qwen2.5-7B, it reached 89.1% success, about 8 percentage points above GRPO’s 81.2% and slightly above StepOPSD’s 88.4%. On Search-QA with Qwen2.5-3B, accuracy was 46.7%, compared with 36.4% for GRPO and 43.4% for SDAR, a self-distillation baseline. The method was not uniformly better: on WebShop’s exact-completion rate with Qwen2.5-7B, AgentOPSD’s 79.7% trailed SDAR’s 82.8%, although it posted the best WebShop score (90.2 versus 80.9 for GRPO and 89.4 for SDAR). The authors report that AgentOPSD beat GRPO+OPSD, Skill-SD, and RLSD on all eight aggregate comparisons, and SDAR on six of eight.

The advantage widened with task horizon. Regressing per-sub-task ALFWorld success on the mean turns of successful episodes, AgentOPSD lost 0.54 success points per extra turn, versus 2.91 for GRPO and 3.59 for RLSD. Ablations support the mechanism: replacing the recursive belief revision with the raw local gap lowered ALFWorld success from 89.1% to 82.8%, and switching from turn-level to token-level accumulation lowered it to 85.9%.

The authors caution that the belief state is a relative support signal, not a calibrated probability, and the method depends on a training-only skill bank retrieved by keyword matching. In domains without such curated skills, the evidence proxy would need another source. Still, the results suggest that self-distillation signals become more useful when treated as evidence that accumulates over time, rather than as isolated scores—an idea that could make agentic reinforcement learning more robust in long-horizon interactive settings, though it is not a general solution to credit assignment.