AI Papers Reader

Personalized digests of latest AI research

View on GitHub

AI Models Struggle with Library Version Incompatibilities, New Benchmark Reveals

New York, NY – The rapid evolution of Python libraries, a cornerstone of modern software development, poses a significant challenge for AI-powered code generation tools. A new benchmark, dubbed GitChameleon, has been introduced to rigorously test the ability of these AI systems to generate code that is not only functional but also compliant with specific library versions. The findings indicate that even state-of-the-art AI models struggle with this task, achieving success rates in the modest 48-51% range.

The GitChameleon benchmark comprises 328 Python code completion problems, each meticulously curated to address documented breaking changes across various popular libraries. Crucially, each problem is accompanied by executable unit tests, allowing for an objective, execution-based evaluation of the generated code. This approach is vital because real-world software development often involves working with specific, sometimes older, library versions due to technical debt or project constraints.

“Contemporary LLMs, agents, and coding assistants are increasingly integrated into the software development workflow. However, a critical capability that remains under-evaluated is generating code that adheres to specific library versions,” explain the authors. “This version-switching capability is essential for robust development in environments with fixed or legacy dependencies.”

The benchmark design differentiates itself from existing ones by focusing on “in-distribution” (ID) generation for specific library versions that models may have encountered during training, rather than “out-of-distribution” (OOD) scenarios. This allows for a more precise assessment of a model’s ability to handle version constraints directly.

To evaluate GitChameleon’s effectiveness, the researchers tested a diverse range of AI systems, including large language models (LLMs) like GPT-4.5 and Claude 3.7, AI agents, and IDE-integrated coding assistants. The results paint a clear picture: while these tools excel at general code generation, they falter when faced with the specific demands of library version compatibility.

For instance, a problem might require using a function from a specific version of the seaborn library, say version 0.13.0. An AI model might attempt to generate code that uses a parameter, like bw, which was deprecated in that version, instead of the correct bw_method or bw_adjust parameters. GitChameleon’s tests would flag this as an error, penalizing the model for its inability to adhere to the specified version’s API.

The study found that while techniques like “self-debugging” – where models attempt to correct their own errors based on feedback – showed improvements, they were not enough to overcome the fundamental challenges of version-specific code generation. The performance differences were also observed to be more pronounced when dealing with newer or less common libraries, suggesting that models might be better trained on widely adopted libraries with extensive documentation.

The researchers hope that GitChameleon will spur the development of more adaptable and dependable AI code generation methods capable of navigating the complexities of evolving software ecosystems. The dataset and evaluation code are publicly available, inviting further research and development in this critical area.