AI Papers Reader

Personalized digests of latest AI research

View on GitHub

When Every Answer Is Wrong, a Teacher Can Help — If Chosen Carefully

Reinforcement learning can sharpen large language models for math and code, but it sometimes has nothing to learn from. A widely used method, Group Relative Policy Optimization (GRPO), samples several answers to a prompt and computes each response’s advantage relative to the group average. If all sampled answers are wrong, the advantage for every token becomes zero and the model receives no training signal. Researchers at Tianjin University and Meituan Longcat propose a method called RSTG that recovers this lost signal by borrowing targeted guidance from a stronger teacher model.

The natural remedy, on-policy distillation (OPD), has the student generate responses and then trains it to match the teacher’s per-token probabilities, producing a dense learning signal. But combining GRPO and OPD naively hurts performance. The team identifies three causes: not all prompts benefit from distillation; students can converge to the teacher too quickly and lose the exploration RL needs; and OPD’s token-level advantages are mostly negative, penalizing the student more than guiding it. That asymmetry was especially severe on all-wrong prompts, where advantages stayed more negative throughout training than on all-correct prompts.

RSTG applies distillation only where it is useful. It targets “negative zero-variance prompts,” prompts where all eight student rollouts fail, and weights each prompt’s distillation signal by the teacher’s confidence, measured as the fraction of eight teacher rollouts that are correct. At the token level, it restricts gradient updates to tokens with high student entropy or large teacher–student divergence, using a composite Soft-OR score. It also runs supervised fine-tuning on correct teacher-generated answers for those hard prompts, adding positive gradients the RL branch cannot provide.

The selection step matters. Starting from 57,000 math prompts, OPD on the 9,000 prompts the student failed beat OPD on the full set. Narrowing to the roughly 2,000 prompts where the teacher was also perfect — 3.63% of the data — worked best.

Across three student–teacher pairs, RSTG outperformed naive GRPO+OPD by 0.89 to 4.02 percentage points on math benchmarks and by 1.97 to 3.05 percentage points on code benchmarks; these are percentage-point changes in accuracy, not relative gains. With a Qwen3-1.7B-Instruct student and Qwen3-4B-Instruct-2507 teacher, average math accuracy rose from 51.37% to 55.39%. RSTG also slowed convergence toward the teacher: at step 200, its overlap with the teacher’s likely next tokens was 65.81%, versus 67.6% for the naive combination and 69.7% for OPD alone.

The authors caution that the method assumes a meaningful capability gap; gains shrank when student and teacher were both 4B models. They tested only math and code, not agentic tasks, and RSTG added about 12 hours of compute on eight A100 GPUs over a 550-step math run. The largest student tested was 4B, so scaling remains an open question.

The work offers a more principled way to combine a teacher’s knowledge with reinforcement learning’s exploration: use the teacher where the student is stuck, but do not let it dominate. The results are consistent across model families, though they do not prove the recipe will transfer to every setting.