Step-Controlled DPO Gives LLMs Stepwise Supervision for Expert Mathematical Reasoning
Artificial intelligence researchers have introduced a novel technique, Step-Controlled Direct Preference Optimization (SCDPO), designed to significantly enhance the mathematical reasoning abilities of large language models (LLMs). This method overcomes a fundamental limitation of standard alignment techniques by providing automatic, granular feedback on where a model’s reasoning goes wrong, step-by-step.
Direct Preference Optimization (DPO), a popular method for fine-tuning LLMs, traditionally relies on comparing a preferred, correct output (the full solution) against a dispreferred, incorrect output. However, in complex, multi-step math problems, an incorrect solution may still contain many correct intermediate steps. Standard DPO penalizes the entire sequence, failing to isolate the specific error.
SCDPO, proposed by researchers from the Chinese University of Hong Kong, addresses this by automatically generating high-quality dispreferred training samples with surgical precision.
The process starts with a base LLM that has been fine-tuned on question-solution pairs. To create an error-annotated dispreferred sample, the system takes a known correct solution, halts the generation process at a random, correct intermediate step, and then forces the model to continue generating the rest of the steps under increasing levels of “temperature” (a hyperparameter that introduces randomness).
This intentional injection of instability causes the model to make an error immediately following the correct prefix. Crucially, the preferred and dispreferred solutions are identical up to the error step.
This stepwise supervision offers superior credit assignment compared to naive DPO. For instance, consider a word problem that requires understanding the phrase “4 less than a dozen.” A model might incorrectly interpret this as $4 \times (12 - 4)$. While a standard DPO model would penalize the final wrong answer, SCDPO specifically highlights the tokens corresponding to the faulty mathematical interpretation, teaching the model precisely which part of the reasoning chain must be avoided.
Similarly, in an advanced number theory problem, if the model incorrectly decides to sum terms instead of multiply terms to find a remainder, SCDPO’s detailed supervision isolates the flawed operation token-by-token, rather than merely marking the whole solution as a failure.
The empirical results demonstrate SCDPO’s effectiveness across multiple models and benchmarks. When applied to three different 7-billion parameter (7B) Mistral models, SCDPO consistently outperformed standard DPO, even when DPO was given an equal or greater amount of total training data.
The most impressive results came from scaling up the approach. SCDPO was used to fine-tune the InternLM2-20B model on extensive English and Chinese math datasets (GSM8K, MATH, APE210K, etc.). The resulting model achieved scores of 88.5% on the grade-school level GSM8K dataset and 58.1% on the high-school and competition-level MATH dataset. This performance rivals or surpasses all other publicly available open-source LLMs in these mathematical reasoning categories, underscoring the potential of detailed stepwise supervision to unlock advanced reasoning capabilities in LLMs.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.