AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Noising a Model’s Weights Gives LLM Reinforcement Learning a New Exploration Lever

Reinforcement learning with verifiable rewards (RLVR) is one of the main ways to sharpen a large language model’s (LLM) reasoning after initial training. In the widely used Group Relative Policy Optimization (GRPO) variant, the model samples several answers per problem and compares their rewards to produce a learning signal. But training stalls whenever all sampled answers to a prompt receive the same reward: the advantage is zero. Raising the sampling temperature to encourage variety does not help much, because temperature flattens probabilities without changing which tokens are most likely. A group of researchers therefore tried perturbing the model’s weights themselves.

The approach, called Perturbed Parameter Policy Optimization (3PO), treats exploration as a parameter-space problem. Using the IVON optimizer—a variational method that learns a Gaussian distribution over the weights—the team maintains a learned distribution over the model’s parameters, then samples one or more slightly altered “noisy” policies at rollout time. Each noisy policy can reorder token probabilities and produce trajectories that the base policy would rarely generate. Three variants are tested. B3PO uses one perturbation per update; M3PO averages gradient updates over several perturbations; C3PO, the best-performing variant, splits each group of 16 rollouts across four different perturbed policies and calculates advantages over the combined, more diverse group.

The concrete failure mode appears with temperature-based action-space exploration. On OLMo-3-1025-7B, a baseline called Polaris increased sampling temperature when entropy collapsed. Because temperature scaling is rank-preserving—it cannot reorder tokens—it mostly generated malformed rollouts with no extractable answer. Polaris lost more zero-advantage groups than it rescued, while the multiple-perturbation 3PO methods continued rescuing such groups throughout training.

Across six math benchmarks, C3PO achieved an average Pass@1—the standard metric for whether the model produces a correct answer—of 44.04% on OLMo-3-1025-7B versus 42.99% for standard GRPO; on Qwen2.5-Math-7B it scored 46.88% versus 45.36%. Gains were largest on the AIME (American Invitational Mathematics Examination) contest sets: up to 4.17 percentage points relative to GRPO on OLMo-3 (AIME 2025) and up to 5.83 percentage points on Qwen2.5-Math (AIME 2026). In code generation, C3PO reached a LiveCodeBench-v6 score of 15.17, and 3PO methods reached GRPO’s final reward within the first half of training. Unlike the single-perturbation variant, the multiple-perturbation methods M3PO and C3PO consistently produced fewer malformed or incorrect rollouts than GRPO, and all 3PO variants ran at near-identical FLOPs per step. The authors hypothesize that parameter-space exploration helps most when the pre-RL model is weak, as on hard math and code tasks.

The method has real costs. Although FLOPs were similar, M3PO and C3PO took about 1.5 times longer per training step than GRPO, a systems overhead the authors attribute to current serving stacks’ inefficient multi-model sampling. The noise level also requires tuning: with too much noise, C3PO collapsed entirely, and the team could not test larger models due to compute limits. Still, the work offers evidence of an orthogonal exploration axis that can be combined with temperature and reward shaping, not a replacement for them.