AI Papers Reader

Personalized digests of latest AI research

View on GitHub

AI Breakthrough: New Method Prevents Reasoning Models from "Forgetting" Basic Skills

San Francisco, CA – Researchers have developed a novel technique to prevent advanced AI language models from losing their foundational knowledge after being trained for complex reasoning tasks. This breakthrough addresses a critical issue known as “capability regression,” where models become proficient in specific areas like math or logic but degrade in general skills such as image recognition or factual recall.

The new method, dubbed RECAP (Replay-Enhanced Capability Preservation), aims to rebalance the learning process. While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved AI’s reasoning abilities, it often comes at the cost of forgetting essential pre-trained knowledge.

The Problem: Forgetting the Basics

Imagine a student who spends months intensely studying advanced calculus. While they might excel at complex equations, they could potentially struggle with basic arithmetic or recalling historical facts they once knew well. Similarly, large language models (LLMs) trained extensively on reasoning tasks, like solving complex math problems or generating code, can suffer from this “forgetting.”

The paper highlights empirical evidence showing that open-source reasoning models often perform worse on general capabilities like visual recognition (e.g., identifying objects in an image) and faithfulness (e.g., sticking to factual information) after being fine-tuned for reasoning. This is because the RLVR process, while rewarding correct reasoning steps, can inadvertently steer the model away from its broader, pre-trained knowledge.

Existing methods to mitigate this, such as using KL divergence to keep the model close to its original state, are often calculated on the current task and don’t guarantee preservation of broader knowledge. Furthermore, simply replaying data from different domains can make it difficult to determine the right balance of focus for each objective.

RECAP: A Smarter Approach to Continued Learning

RECAP tackles this challenge by introducing a dynamic reweighting strategy. Instead of uniformly distributing training focus, RECAP intelligently adjusts priorities based on how well each objective is progressing.

Here’s how it works:

  1. Replaying General Knowledge: RECAP integrates general-purpose data alongside the specific reasoning tasks during the fine-tuning process. This ensures that foundational skills are continuously reinforced.
  2. Dynamic Objective Reweighting: The core innovation lies in how RECAP balances different learning objectives. It monitors the “convergence rate” and “instability” of each objective in short-term windows.
    • Convergence Rate: How quickly a particular skill is being learned or improved.
    • Instability: How much the learning signal for a particular skill fluctuates.

For example, if a model is quickly mastering a specific formatting requirement (leading to high convergence and low instability), RECAP will naturally down-weight its focus on that particular objective. Conversely, if a core reasoning skill is progressing slower or is more volatile, RECAP will allocate more attention to it. This dynamic adjustment allows the model to learn new reasoning skills without letting its general capabilities degrade.

Tangible Benefits: Improved Reasoning and Preserved Skills

The researchers demonstrated RECAP’s effectiveness on Qwen2.5-VL models, showing significant improvements. Not only did RECAP preserve general capabilities, but it also enhanced reasoning performance by enabling more flexible trade-offs among different reward signals.

  • Concrete Example: Imagine an AI being trained to answer questions about images. Without RECAP, it might become excellent at solving visual math problems but forget how to accurately describe the objects in a scene. With RECAP, the AI would still improve its math skills but also maintain its ability to describe the scene, as RECAP would ensure that the “description” learning objective remains active and appropriately weighted.

The study also showed that RECAP-tuned models generated shorter, more concise rationales for their answers, improving inference efficiency without compromising accuracy. This suggests that RECAP not only preserves knowledge but also encourages more efficient and effective reasoning.

This work offers a promising solution to a persistent challenge in AI development, paving the way for more robust and capable AI systems that can learn and adapt without sacrificing their fundamental understanding of the world.