AI Papers Reader

Personalized digests of latest AI research

View on GitHub

CritiQ: AI System Mines Data Quality Criteria From Human Preferences to Improve Language Models

Large language models (LLMs) rely heavily on high-quality training data. Researchers from Fudan University, Shanghai AI Laboratory, East China Normal University, and the University of Science and Technology of China have introduced a novel data selection method called CritiQ, which automatically identifies and extracts data quality criteria from human preferences, significantly reducing the need for expert knowledge and manual effort. The work is published in a paper available on arXiv.

The traditional approaches involve manually crafting heuristics, calculating perplexity using existing models, training classifiers, or employing careful prompt engineering. These methods can be resource-intensive and prone to biases. CritiQ addresses these limitations by using LLM-based agents to distill quality criteria from as few as 30 human-annotated data pairs.

The core of CritiQ is CritiQ Flow, a process consisting of a “manager” agent and “worker” agents. The manager evolves quality criteria while the workers make pairwise judgments based on the criteria. The system leverages a knowledge base of existing data quality criteria, derived from prior research, to bootstrap the process. A final, numerical quality score is then calculated to perform data selection. This verbal criteria derived in this manner is more interpretable and has reusable value.

Here’s how CritiQ works:

  1. Human Annotation (Small Scale): A small set of data pairs (e.g., 30 pairs of code snippets, math problems, or logical statements) are presented to human annotators who indicate which example in each pair exhibits higher quality.

  2. CritiQ Flow – Manager and Workers: The system employs a manager agent, which is implemented using GPT-4, and worker agents, which are implemented using Qwen2.5-72B-Instruct. It uses a knowledge base that it builds from previous work to boost the process. The worker agents perform pairwise comparison under a given criterion. The manager agents generates new criteria and refine them through reflection on worker agents’ performance. The final judgement is done by majority voting.

  3. Quality Scoring: The extracted criteria are then used to train a lightweight scoring model (CritiQ Scorer). This model is a Bradley-Terry model. This model assigns numerical scores to each data point in a much larger corpus based on the learned quality criteria.

  4. Efficient Data Selection: CritiQ Scorer enables efficient data selection, allowing researchers to prioritize high-quality data for training LLMs.

To validate CritiQ, the team performed experiments across code, math, and logic domains. For example, in the code domain, the system might learn criteria like “the code should be well-structured with proper comments,” or “the code shows clear purpose and can accurately solve certain kind of problems”. For the math domain, it might extract “expressions should use precise and consistent notation” from the previous work. CritiQ consistently improved accuracy on human-annotated test sets compared to alternative methods.

Furthermore, continual pre-training experiments using Llama 3.1 models trained on data selected by CritiQ showed improved performance on downstream tasks compared to models trained on uniformly sampled data. This demonstrates that CritiQ can effectively identify and select high-quality data that enhances model performance.

Ablation studies also confirmed the effectiveness of both the knowledge base and the reflection process in CritiQ. Analysis revealed how the criteria evolve during the process and the importance of majority voting.

In conclusion, CritiQ offers an innovative approach to data selection for LLMs by automating the extraction of data quality criteria from human preferences. This significantly reduces the reliance on manual effort and expert knowledge, potentially leading to more robust and generalizable LLMs.