AI Papers Reader

Personalized digests of latest AI research

View on GitHub

THOR: A New Framework for Mathematical Reasoning in Large Language Models

Large Language Models (LLMs) have made significant strides in understanding and generating human language, but they often falter when it comes to precise tasks like mathematical calculations and formal logic. A new research paper introduces THOR (Tool-Integrated Hierarchical Optimization via RL), a novel framework designed to empower LLMs with robust mathematical reasoning capabilities by effectively integrating external tools.

The core of THOR addresses three key challenges: constructing suitable training data, optimizing model performance in a fine-grained manner, and enhancing the accuracy of reasoning during inference.

TIRGen: Crafting Intelligent Training Data

To tackle the data scarcity and quality issues, the researchers developed TIRGen, an actor-critic pipeline for generating high-quality tool-integrated reasoning datasets. Imagine an “Actor” LLM that generates natural language reasoning steps, and a “Critic” LLM that identifies which of these steps can be best handled by a code interpreter. The Critic then converts these steps into executable Python code, runs them, and uses the results to refine the reasoning process. This iterative “think-act-observe” cycle ensures the generated data aligns with the LLM’s own reasoning style and is broadly applicable across different models.

For example, if an LLM is trying to solve a problem like calculating the area of a complex shape, TIRGen would identify a step that involves a trigonometric calculation. The Actor might describe this as “calculate the sine of angle X,” and the Critic would translate this into print(math.sin(X)) in Python, execute it, and then feed the numerical result back into the LLM’s reasoning chain.

Hierarchical Optimization: Learning to Reason and Code

THOR employs a sophisticated hierarchical Reinforcement Learning (RL) strategy. This approach is inspired by a key insight: the success of an intermediate tool call is a strong predictor of the final answer’s correctness. THOR optimizes the LLM at two levels:

  1. Trajectory-level Optimization: This focuses on the overall correctness of the final answer to a mathematical problem, akin to winning the entire game.
  2. Step-level Optimization: This dives deeper, fine-tuning the model’s ability to generate correct and executable code for specific, error-prone steps within the reasoning process. This is like ensuring each individual move in the game is strategically sound.

For instance, when solving a multi-step math problem, trajectory-level optimization rewards the model for arriving at the correct final number. Step-level optimization, however, specifically rewards it for accurately writing and executing Python code for intermediate calculations, such as solving an equation or performing a complex algebraic manipulation.

Self-Correction: Learning from Mistakes on the Fly

During inference, THOR incorporates a self-correction mechanism. If a tool execution fails, the LLM doesn’t just give up. Instead, it uses the feedback from the failed tool call to dynamically revise its reasoning. It can backtrack, re-evaluate its steps, and attempt alternative approaches. This is like a student realizing they made a mistake in a calculation, going back to check their work, and trying a different method if necessary.

An example of this could be an LLM attempting to solve a geometry problem. It might try to use a calculator tool to find the length of a side using the Pythagorean theorem. If the tool returns an error (perhaps due to incorrect input), THOR would prompt the LLM to review its input and potentially try a different formula or approach.

Performance and Generalization

The paper demonstrates that THOR achieves state-of-the-art results on various mathematical benchmarks for models of comparable size. Crucially, THOR exhibits strong generalization capabilities, performing effectively on both models designed for reasoning and those that are not. Furthermore, THOR shows impressive improvements on code generation benchmarks, validating its robustness and versatility. The framework also demonstrates computational efficiency, requiring fewer tokens during inference compared to baseline models.

In summary, THOR represents a significant advancement in enabling LLMs to tackle complex mathematical problems by seamlessly integrating external tools, optimizing their reasoning process hierarchically, and equipping them with a self-correction capability.