AI Papers Reader

Personalized digests of latest AI research

View on GitHub

A Simple "Try Again" Command Unlocks Deeper Multi-Turn Reasoning in LLMs

Large Reasoning Models (LRMs), lauded for their ability to solve complex problems in a single attempt, have a surprising flaw: they often fail spectacularly when given feedback and asked to try again. New research from Imperial College London, Northwestern University, the University of Washington, and IBM Research AI addresses this critical weakness by introducing a novel reinforcement learning (RL) framework called Unary Feedback as Observation (UFO).

Current RL methods typically train LLMs using a single-turn paradigm, optimizing them to produce a correct answer immediately. However, the researchers found that models trained this way become “effective solvers but poor revisers.” When a user prompts a single-turn model with an incorrect answer and gives simple feedback like, “Please try again,” the model stubbornly repeats its original mistake. In the study, single-turn trained models generated the exact same wrong answer across five consecutive turns in 70% of failure cases.

The persistence of these models stems from a low-entropy output distribution created during single-turn RL training, making them mathematically guaranteed to have a high repetition rate.

The UFO framework resolves this by transforming traditional, static single-turn datasets into interactive, multi-turn episodes using only minimal feedback. Crucially, UFO uses unary (negative-only) feedback—a generic signal like “Incorrect. Please think again.” is appended to the dialogue history whenever the model fails. This minimal observation forces the LRM to condition its next response on the full history of failed attempts, encouraging genuine self-reflection and the exploration of alternative reasoning paths.

When solving a math problem—for instance, one requiring the model to find a missing variable X by summing letter values in a pattern—the contrast between models is stark. A standard single-turn RL model might try to solve the problem, get it wrong, and then simply output the same incorrect reasoning and answer four more times. In contrast, a UFO-trained model receives the minimal “Incorrect. Please think again.” signal and proceeds to systematically revise its assumptions and calculation across subsequent turns until it reaches the correct answer.

The results show that RL training augmented with UFO effectively unlocks these interactive reasoning capabilities. Experiments across diverse domains, including math, STEM, and general knowledge tasks, demonstrated that UFO improves multi-turn reasoning success rates by up to 14% over conventional single-turn baselines, while preserving strong single-turn performance.

To further encourage efficiency and thoughtful responses, the researchers enhanced the RL reward mechanism with two guiding principles: minimality (solving the problem in the fewest turns) and diversity (exploring varied strategies). This was achieved through turn-wise reward decay, which favors early success, and an answer repetition penalty, which actively penalizes repeated outputs.

This reward shaping successfully guided the models to produce more systematic and careful answers, boosting the proportion of unique and effective answers generated during subsequent attempts from 80% to over 90%.

The UFO framework provides a simple, lightweight, and scalable solution for integrating conversational feedback into existing RL pipelines, enabling LLMs to robustly generalize their reasoning skills from single-shot tasks to complex, real-world interactive problem-solving scenarios.