Data Mixing Agent: New AI Learns Optimal Recipe to Prevent LLM Forgetting
Large Language Models (LLMs) often undergo continual pre-training to specialize in complex areas like mathematical reasoning or code generation. While effective, this process carries a high risk of ācatastrophic forgettingāāwhere the model loses its original, broad general knowledge as it absorbs new, specialized skills.
Researchers from the University of Manchester and Microsoft Research have introduced the Data Mixing Agent (DMA), an innovative solution that uses reinforcement learning (RL) to automatically determine the optimal ārecipeā for mixing training data, effectively balancing specialization and general knowledge preservation. DMA is the first model-based, end-to-end framework designed to learn these domain re-weighting strategies dynamically.
Moving Beyond Human Heuristics
Traditionally, balancing source data (the general knowledge dataset) and target data (the specialization dataset) relied on human-defined heuristicsārules of thumb established through empirical testing. For instance, experts might manually decide that a certain ratio of āScience domainā data must be included to maintain performance on general knowledge tests.
The Data Mixing Agent replaces this manual designation with an autonomous system. It frames the data mixing challenge as a Markov Decision Process, allowing an AI agent to learn sophisticated mixing strategies by observing the outcomes of millions of simulated ādata mixing trajectories.ā
The agent is trained offline using Conservative Q-Learning (CQL), a form of reinforcement learning. Essentially, the DMA acts as a strategic chef, receiving feedback on whether its current mixture of data domains (like āHobbies & Leisureā versus āFinanceā) improves or degrades the performance of the target LLM on a wide range of benchmarks.
Superior Balance and Generalization
When tested on continual pre-training for math reasoning, the Data Mixing Agent demonstrated superior performance across both general knowledge and specialized benchmarks, confirming its ability to alleviate catastrophic forgetting.
In experiments, DMA significantly outperformed strong baseline methods like RegMix, achieving an average performance improvement of 3.02% across general benchmarks and four dedicated math reasoning benchmarks. This proves the agent can enhance targeted capabilities without sacrificing the LLMās general skills.
Crucially, the learned heuristics proved highly generalizable. An agent trained specifically on the math reasoning domain could be immediately applied to guide pre-training for the entirely new domain of code generation, achieving leading performance without any retraining. This suggests the agent learns fundamental, data-agnostic mixing principles rather than merely memorizing task-specific data weights.
The agent also operates with remarkable efficiency. Analysis of its data mixture trajectory revealed it can achieve superior results while relying on 2.14 billion fewer tokens from the source field than baselines, substantially reducing the computational cost of future pre-training efforts.
A Three-Stage Strategy
Further insights into the agentās actions show it adopts a well-aligned, conservative three-stage strategy, demonstrating learned intuition:
- Early Warm-up: The agent initially prioritizes source field data to stabilize the target model.
- Mid-training: It rapidly increases target field data use to accelerate performance gains in the new, specialized capability.
- Final Stage: The agent gradually reintroduces more source field data, stabilizing the distribution around an optimal balance point, ensuring long-term skill retention.
By treating the training recipe as an optimizable parameter guided by an intelligent agent, the Data Mixing Agent offers a highly efficient and effective path forward for adapting LLMs to new tasks while preserving their foundational intelligence.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.