AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Backprop-Free Tuning Keeps LLM Reasoning Broader Than Standard RL, Study Finds

When developers refine large language models to solve math or science problems, the standard reinforcement-learning recipe can make the model better on a first attempt while quietly making it less likely to find a correct answer over many attempts. A new study argues that an alternative, backpropagation-free method called evolution strategies avoids that narrowing, and may preserve access to a wider range of correct reasoning paths.

Evolution strategies, or ES, tune a model by perturbing its parameters, running the perturbed versions on a batch of prompts, and then averaging the perturbations weighted by the rewards those versions received. It needs no gradient computation. The researchers compared it with Group Relative Policy Optimization, or GRPO, a widely used method that samples several responses from the same policy and backpropagates a token-level objective based on their relative rewards.

The authors’ central claim is that ES explores more broadly. Because ES evaluates a population of slightly different policies, they argue, the population can contain members with different chances of producing a correct answer; sampling across those members increases the probability that at least one response succeeds. The paper formalizes this using a diversity measure called Jensen–Shannon divergence, projected through a correctness verifier.

A concrete example shows the practical difference. After training Qwen2.5-1.5B-Instruct on GSM8K, the researchers tested it on GPQA, a graduate-level science benchmark. GRPO raised Pass@1, the chance a single response is correct, by 3.90 percentage points relative to the base model, but Pass@16 and Pass@32 — the chance that at least one of 16 or 32 sampled responses is correct — fell by 1.77 and 2.02 percentage points. ES improved all three metrics, by 2.02, 4.91, and 5.05 percentage points.

Across models trained on GSM8K, ES improved average Pass@1, Pass@16, and Pass@32 over the base model, and it beat GRPO on average Pass@16 and Pass@32. GRPO achieved larger Pass@1 gains, but it fell below the base model on both large-K metrics in 15 of 18 comparisons. The same pattern appeared with DeepSeek-R1-Distill-Qwen-1.5B trained on the harder DeepScaleR dataset.

ES also moved far more in parameter space: final ES models were 40.7 to 44.1 times farther from their starting weights than GRPO counterparts, measured by relative Euclidean distance. Yet the authors found that most of that movement was functionally concentrated. In their thresholding experiments, between 77.6% and 93.0% of nonzero updates had small magnitudes; resetting those small updates to zero largely preserved task accuracy. They also reported that ES generally retained held-out performance, with average Pass@32 changes on held-out tasks positive under ES but negative under GRPO. This challenges earlier claims that large parameter drift under ES necessarily causes catastrophic forgetting.

The study includes caveats. ES improved Pass@1 less than GRPO did, so the authors tested sequential training orders, ES→GRPO and GRPO→ES, to combine both strengths; the better order depended on the task. They also note that ES can still lose performance on some held-out tasks, and that its effects over longer, multi-task continual-learning horizons remain unresolved. They found that z-score reward normalization is important, and that larger models need smaller ES populations.

The broader significance is not that ES outperforms GRPO outright, but that it occupies a different point on the accuracy-versus-coverage frontier. That distinction could make ES a useful tool for applications that rely on repeated sampling — or a stable partner for conventional reinforcement learning.