AI Models Learn to 'Think' Independently Using Massive Unlabeled Data
Reinforcement Learning on Pre-Training Data (RLPT) promises significant reasoning gains by eliminating reliance on human feedback.
A team of researchers from Tencent and other institutions has introduced a novel training paradigm for large language models (LLMs) called Reinforcement Learning on Pre-Training data (RLPT). This breakthrough method applies powerful reinforcement learning (RL) techniques directly to massive, unlabeled web corpora, effectively teaching models to self-supervise their own reasoning processes.
The move addresses a major scalability bottleneck inherent in current state-of-the-art methods, such as Reinforcement Learning from Human Feedback (RLHF) and other RL variants, which are constrained by the cost and scarcity of high-quality human annotations. RLPT replaces human guidance with a self-supervised “next-segment reasoning” objective, allowing RL to scale indefinitely on existing pre-training data.
In contrast to traditional supervised pre-training, where models predict the next token (word) one by one, RLPT focuses on predicting the next segment of text—a semantically coherent unit like a full sentence or a reasoning step. This forces the model to learn the underlying logic of the text, not just its sequence.
The framework incorporates two primary training tasks: Autoregressive Segment Reasoning (ASR), which predicts the subsequent segment given the preceding context (similar to forecasting the next step in an argument), and Middle Segment Reasoning (MSR), which predicts a masked segment given both preceding and following context (useful for inferring a missing step in a complex proof).
The critical innovation lies in the reward signal. To avoid rigid word-for-word matching, the system employs a generative reward model ($G_{rm}$) that grants a score of 1 if the predicted segment is a semantically consistent prefix of the ground-truth text. For example, if the real text is, “First, we calculate the area of the circle and then double it,” the model receives full reward for predicting, “We calculate the area of the circle,” because it successfully completed the initial step and maintained semantic flow, even if it didn’t generate the entire reference segment. This relaxed “prefix reward” provides a stable and effective self-supervision signal.
Experiments across various LLMs, including Llama and Qwen3 architectures, demonstrate substantial improvements in both general-domain and mathematical reasoning. Applied to Qwen3-4B-Base, RLPT yielded an absolute gain of 8.1 points on the demanding GPQA-Diamond benchmark and a 6.0-point gain on KOR-Bench.
The gains were particularly striking in mathematical problem-solving. On the highly challenging AIME24 benchmark, RLPT improved Pass@1 accuracy by 6.6 points. Furthermore, when RLPT was used as a robust initialization for subsequent, more complex Reinforcement Learning with Verifiable Rewards (RLVR), performance was boosted even further, achieving an additional 2.3-point gain on AIME24 Pass@1.
Crucially, researchers confirmed that RLPT performance follows a favorable power-law scaling trend with increased training compute, empirically establishing a clear path for continued performance gains as computational resources expand. By enabling LLMs to learn complex reasoning from the vast ocean of unlabeled data, RLPT lays a strong foundation for future, highly scalable AI models.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.