AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Teaching Large Language Models When to Stop and Say “I Can’t”

When a large language model is asked something it cannot actually solve, it often keeps going anyway—producing long, confident-looking chains of reasoning that are subtly wrong. The researchers call this “futile reasoning,” and argue it is a serious reliability problem: users can mistake fluent but incorrect derivations for valid answers. In a new preprint, researchers from the Chinese Academy of Sciences and Tencent describe CaRL, a training method designed to make models refuse when they hit their capability limits.

The team tested models including Qwen3-8B, Qwen3-32B, gpt-oss-120b, and DeepSeek-V3.2 on Countdown puzzles, in which a model must combine a set of numbers with arithmetic operations to reach a target. Across all difficulty levels, the untuned baseline models never refused; even when explicitly instructed to admit ignorance, more than 80% of failures on the hardest puzzles still consisted of futile reasoning rather than refusal. The dominant failure mode was specious reasoning: outputs that look superficially valid but contain subtle errors, such as misused numbers or false arithmetic. In one described example, DeepSeek-R1 was given numbers [1,5,6,8,9] and asked to make 0.3. It produced an expression that actually equals 0.6 and presented it as a solution.

CaRL, short for Capability-aligned Reinforcement Learning, changes what the model is rewarded for during training. Instead of treating every wrong answer and every refusal as equally bad, CaRL gives +1 for a correct answer, 0 for a valid refusal, and -1 for an incorrect answer. That creates a clear incentive to quit rather than hallucinate. Because models rarely generate refusals on their own, the method also uses “hindsight refusal augmentation”: failed reasoning traces are converted into refusal examples by keeping the attempted steps, adding a phrase like “Sorry, I cannot solve this problem. Here is how far I got,” and asking the model to summarize what it tried.

The main results show large reductions in futile reasoning. On Countdown tasks, CaRL cut the futile rate from 65.5% to 7.0% for Qwen3-8B and from 78.6% to 1.0% for Qwen3-14B—drops of 58.5 and 77.6 percentage points. It also improved a reliability score that counts correct answers fully and refusals half, while Countdown accuracy was maintained or improved and accuracy on AIME 2024 and GPQA moved less than 2 percentage points. On out-of-distribution Sudoku—a task not used in training—futile reasoning fell from 89.4% to 43.0% on the 8B model and from 80.5% to 36.0% on the 14B model. But there was a cost: accuracy on Sudoku for the 14B model fell 5.9 percentage points, from 50.75% to 44.87%, showing that teaching refusal can sometimes reduce willingness to attempt hard problems.

The authors caution that their experiments used Countdown and Sudoku, pure algorithmic tasks with no external knowledge, so real-world tasks where knowledge and reasoning boundaries mix remain an open question. Still, they argue that capability-aligned training could make LLMs safer and cheaper by reducing the computational waste and misleading output caused by futile reasoning. The study is a step, not a final fix: even in their tests, the method did not eliminate over-refusal or all futile attempts, and its generalization to broader domains is not yet established.