AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Long Code Arena Challenges LLMs to Master Project-Scale Software Tasks

As Large Language Models (LLMs) grow in complexity and context capacity, new benchmarks are needed to evaluate their practical utility in real-world software engineering. Researchers from JetBrains Research and Delft University of Technology have introduced the Long Code Arena, a comprehensive suite of six benchmarks designed to test LLMs on tasks requiring project-wide context, moving beyond the limits of short code snippets.

The initiative addresses a major limitation in current Machine Learning for Software Engineering (ML4SE) evaluation. Popular datasets like HumanEval typically require models to process less than 1,000 tokens to generate a short function. However, modern LLMs can handle context windows spanning tens of thousands or even millions of tokens. The Long Code Arena aims to evaluate how well models can utilize information scattered across an entire codebase, simulating real-life development scenarios.

The new benchmark suite focuses on complex, multi-file tasks:

  1. Library-based Code Generation: Models are instructed to generate a new file that solves a task by heavily utilizing existing APIs within a specific software library. For instance, instead of creating a standalone script, the model must read the library’s source code (averaging 1.4 million characters) to correctly integrate existing classes and methods.
  2. CI Builds Repair: The model must generate a patch to fix a failing Continuous Integration (CI) workflow. This requires analyzing long log files (median 6.5K symbols) and understanding project dependencies to implement the fix. Success is measured by running the generated patch via GitHub Actions.
  3. Project-Level Code Completion: Going beyond single-file completion, models must generate a line of code using relevant context from other files in the repository snapshot. The task is categorized by where the crucial context originates (e.g., from the current file, other files in the project, or files added in the same commit).
  4. Commit Message Generation: Models must generate a concise, human-readable summary for large, multi-file code changes, testing their ability to grasp the overall intent of complex diffs that can span tens of thousands of characters.
  5. Bug Localization: Given a natural language bug description (an issue report), the model must identify the one to five files in a repository snapshot (which averages 1,000 files) that need modification to resolve the issue.
  6. Module Summarization: The task is to generate high-level textual documentation—such as a tutorial or guide—for a project module, based on its source code and intended function.

Initial baseline evaluations using leading models like GPT-4 and open-source alternatives such as Mistral and CodeLlama confirm the challenge posed by the new tasks.

On the Library-based Code Generation benchmark, GPT-4 achieved the best API Recall score at 37%, a metric reflecting how successfully it used the target library’s specific functions. For the demanding CI Builds Repair task, GPT-3.5 successfully fixed only 17% of the samples, demonstrating that models struggle to translate long log context into accurate, actionable code patches.

The researchers conclude that while these complex, project-level challenges are “within reach,” they are “far from solved.” The Long Code Arena is now publicly available on HuggingFace Spaces, providing the necessary datasets, evaluation tools, and baseline results to stimulate focused research into long-context and retrieval-augmented strategies essential for the next generation of AI-enabled software engineering tools.