AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Test-Time Training Without Labels: Asymmetric Updates Let LLMs Learn From Their Own Wrong Answers

Large language models often improve at math by training on solutions with known answers. But in test-time training—adapting to unlabeled problems at inference—no ground truth exists. A common workaround, majority-vote pseudo-labels, is fragile: if the majority answer is wrong, every training signal built on it can be corrupted. Researchers at Zhejiang University and Alibaba Group report an approach that exploits this weakness rather than avoiding it.

The method, Test-Time Policy Optimization (TTPO), starts by sampling 64 solutions per problem and taking the most frequent answer as a pseudo-label. Rollouts that match the label are treated as positive; those that don’t are negative. When Qwen3-1.7B trained on AIME 2026 problems, the authors observed, pseudo-labels were wrong for about 85% of prompts, yet about 79% of rollouts disagreeing with the label were also wrong. Because of that asymmetry, they argue, penalizing disagreement is usually safe even when the label itself is incorrect, whereas distilling toward that label would spread the error token-by-token.

TTPO applies each signal where it is reliable. Positive rollouts are distilled with on-policy self-distillation (OPSD): the same model, prompted with the pseudo-label as a reference, acts as a teacher for its own unassisted student. Negative rollouts receive penalties from GRPO, a reinforcement-learning update that scores trajectories relative to one another. Token-level selection sharpens both arms. In a geometry example from the paper, a positive solution that reached the correct answer, 7, was weighted to concentrate on the insight “the extension beyond A is the line x=0 … going downwards” rather than deterministic coordinates. In a negative solution that arrived at sqrt(397), masking penalized the confident but wrong claim “which is vertical” while sparing locally correct arithmetic such as “(s,s)” and “=6.”

Results span five competition-level benchmarks. On OpenThoughts training data—where label-dependent OPSD used ground truth but TTPO did not—TTPO matched or exceeded OPSD, averaging 40.1% vs 39.7% on Qwen3-1.7B, 58.6% vs 58.4% on Qwen3-4B, and 62.6% vs 61.7% on Qwen3-8B. In pure test-time training, with no labels for any method, Qwen3-1.7B rose from 38.0% to 45.2% average accuracy across AIME 2026, HMMT 2026, and BRUMO 2025, ahead of the label-free reinforcement-learning baseline TTRL (40.2%) and an OPSD test-time baseline (41.9%). Separately, in the OpenThoughts setting, evaluating with thinking mode disabled, TTPO produced gains of 25.2, 30.6, and 36.4 percentage points across scales, several times larger than OPSD’s 7.1, 5.8, and 3.5 points.

The authors note limitations. TTPO depends on majority-vote quality; if too few samples are drawn or no rollout is correct, supervision degrades. The experiments also cover only math problems with extractable final answers, leaving open whether the approach extends to code generation or open-ended reasoning.

Even so, the work suggests that noisy self-supervision is not an obstacle to be eliminated but a signal to be routed carefully. By penalizing what the model disagrees with and distilling what it can reproduce, TTPO offers a label-free path to improving reasoning at the moment it is needed.