AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Harmonizing Reasoning Process and Outcome with PROF: A New Approach to Reinforcement Learning for Mathematical Tasks

Researchers have developed a novel method called the Process Consistency Filter (PROF) to improve the reasoning abilities of AI models in complex mathematical tasks. This new approach aims to overcome limitations in current reinforcement learning techniques, which often struggle to balance the accuracy of final answers with the quality of the reasoning process used to arrive at them.

Current methods, known as Outcome Reward Models (ORMs), provide a reward signal based solely on whether the final answer is correct. While effective for simple verification, ORMs are too coarse-grained to detect subtle flaws in an AI’s reasoning. For instance, an AI might arrive at the correct answer through a fundamentally flawed logical process, which an ORM would deem correct. This can lead to “noisy and misleading gradients” during training, hindering the AI’s overall reasoning quality.

To address this, researchers also explored Process Reward Models (PRMs), which offer more granular feedback by evaluating each intermediate step of an AI’s reasoning. However, PRMs can be prone to inaccuracies and susceptible to “reward hacking,” where the AI learns to exploit the PRM’s scoring system rather than genuinely improving its reasoning.

PROF offers a solution by harmonizing these two approaches. It acts as a data curation strategy that filters AI-generated reasoning steps based on their consistency between the fine-grained PRM signals and the coarse-grained ORM outcomes. Instead of simply blending PRM and ORM rewards, PROF intelligently selects training samples. It prioritizes correct responses that demonstrate high consistency in their reasoning process while also retaining incorrect responses that show promise in their intermediate steps. This selective filtering aims to eliminate conflicting and noisy gradients, leading to more stable and effective training.

A key innovation of PROF is its ability to distinguish between truly correct reasoning and reasoning that coincidentally reaches the correct answer through flawed logic. The paper highlights an example of a coin weighing puzzle where an AI provided the correct answer but through an unbalanced and fundamentally invalid weighing strategy. Standard ORMs would miss this flaw, but PROF’s process consistency check can identify such inconsistencies.

Experiments conducted by the researchers demonstrated that PROF significantly improves the final accuracy of AI models on mathematical reasoning benchmarks, surpassing simpler blending approaches by over 4%. Crucially, PROF also enhances the quality of the intermediate reasoning steps, making the AI’s thought process more detailed, logical, and easier to verify. The authors emphasize that PROF is a modular framework that can be integrated with various reinforcement learning algorithms, making it a versatile tool for advancing AI’s mathematical reasoning capabilities.