AI Papers Reader

Personalized digests of latest AI research

View on GitHub

LLMs Can Think Now: New Technique Allows AI to Reason and Plan Before Answering

Large language models (LLMs) are revolutionizing the way we interact with computers. They can summarize texts, write stories, answer questions, and even generate code, but they often struggle with complex instructions that require reasoning or planning. To bridge this gap, researchers at Meta FAIR have developed a novel training method that enables LLMs to “think” before responding.

The key insight is that, just as humans often think through a problem before providing an answer, LLMs can benefit from generating internal thought processes. This Thinking LLM approach allows the model to explore the space of possible solutions and plan its response strategically, leading to more nuanced and accurate answers.

The researchers call this new technique Thought Preference Optimization (TPO). It leverages a judge model to assess the quality of the LLM’s responses, allowing it to indirectly learn how to generate better thoughts. This iterative process starts with a standard instruction-tuned LLM and trains it to generate thoughts in a specific format before producing a response. The model then uses its internal thought processes to produce a final response, which is then evaluated by the judge. This feedback is used to refine the LLM’s thought processes, iteratively improving the quality of its responses.

The results of this research are impressive. The Thinking LLMs trained using TPO outperform traditional LLMs on a variety of benchmark tasks, including AlpacaEval and Arena-Hard. The model even demonstrates notable gains on categories that are not typically considered to be reasoning-heavy, such as general knowledge, marketing, and health. This suggests that Thinking LLMs have the potential to be more broadly applicable than previous reasoning-focused approaches.

However, the researchers also acknowledge certain limitations. While TPO shows significant promise, further research is needed to address the challenges of teaching LLMs to think in a more nuanced and flexible way. For example, the current system doesn’t allow for explicit control over the length of the thought process, and the model’s ability to generate optimal thoughts might vary depending on the specific task. Despite these limitations, this groundbreaking research opens the door for a new generation of AI that can think and reason, paving the way for more sophisticated and human-like interactions with computers.