AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Here's a markdown file summarizing the research paper:

Code LLMs Need to Get Better at Understanding Human Preferences: A New Benchmark Shows the Gap

Large language models (LLMs) are rapidly advancing their capabilities in code generation. However, a new research paper, “Evaluating and Aligning CodeLLMs on Human Preference,” reveals a significant gap between the performance of these models on traditional code benchmarks and their ability to satisfy actual human preferences when generating code. This study introduces a new benchmark, CodeArena, designed to address this gap and facilitate improved alignment between LLMs and human users.

The Problem: Beyond Correctness

Existing code benchmarks primarily focus on whether the generated code is correct, typically by testing the output against predefined test cases. While correctness is essential, it doesn’t capture the nuances of human preferences in real-world coding scenarios. For instance, a user might prefer a code snippet that is well-documented, easy to understand, efficient, or follows specific coding style guidelines, even if a less readable or less efficient solution is technically correct. The paper highlights this with a comparison (Figure 1) between GPT-4 and Qwen2.5-Coder’s response to a simple “quick sort” request. GPT-4 provides a more detailed and user-friendly answer which includes algorithm explanation and detailed comments, while Qwen2.5-Coder offers only a concise code snippet, ignoring the user’s likely preference for understanding how the code works.

CodeArena: A Human-Centric Benchmark

To bridge this gap, the researchers introduce CodeArena, a novel benchmark consisting of 397 high-quality coding tasks sourced from real-world user queries. These tasks span 40 categories and 44 programming languages, reflecting the diversity of actual coding challenges faced by developers. The tasks aren’t just simple coding exercises; they involve practical scenarios requiring more than just code correctness; factors like readability, clarity and overall user experience come into play. Each sample includes multiple answers generated by advanced models such as GPT4 for reference. These samples are rigorously curated to eliminate biases and ensure high-quality evaluation.

SynCode-Instruct: Scaling Synthetic Instruction Data

The paper also introduces SynCode-Instruct, a massive synthetic instruction corpus containing nearly 20 billion tokens. This corpus is generated by a two-stage process; first, using Qwen2.5 to generate a large volume of synthetic instructions and, second, fine-tuning those with higher quality data from GPT-4. This large dataset then is used to fine-tune a new model, Qwen2.5-SynCoder which serves as a strong baseline for comparison.

Findings: A Performance Gap

By evaluating over 40 LLMs on CodeArena, the study reveals significant performance differences compared to traditional execution-based benchmarks like HumanEval and MultiPL-E. Open-source models, while competitive in terms of code correctness, often lag behind closed-source models in satisfying human preferences as measured by CodeArena. Qwen2.5-SynCoder, trained on the large synthetic dataset, shows considerable improvement, indicating the potential of large-scale synthetic instruction data for aligning models with human preferences. This highlights the crucial role of human preference alignment in building truly effective code LLMs.

Implications:

This research emphasizes that evaluating LLMs based solely on code correctness is insufficient. CodeArena provides a valuable new tool for assessing and improving the human-centric aspects of code generation, pushing the field to develop LLMs that are not only correct but also helpful, user-friendly, and tailored to real-world coding needs. The study’s findings also underscore the significance of high-quality, diverse training data in bridging the gap between current code LLM capabilities and satisfying human expectations.