IBM’s Granite Code Models Achieve Massive 128K Context Window, Transforming Code LLM Capabilities
IBM Research has announced a major breakthrough in foundation models for software development, successfully scaling its Granite 3B and 8B code models to support context windows of up to 128,000 tokens. This dramatic increase—a leap from the original models’ 2K or 4K context limits—equips the open-source Granite family with capabilities previously restricted largely to proprietary large language models (LLMs).
For developers, this extended context window fundamentally changes how code models function. A traditional model might only “see” a single file or a few hundred lines of code at a time, limiting its ability to understand complex interactions. The 128K context allows the scaled Granite models to process an entire software repository, including documentation, build files, and inter-file dependencies, all in a single query.
To achieve this feat without complex architectural changes, IBM employed a lightweight strategy centered on “continual pretraining” and advanced data engineering. The researchers gradually adjusted the RoPE (Rotary Position Embedding) base frequency, a crucial technique used by LLMs to handle longer sequences, while exposing the models to specially prepared long-context data.
A key innovation was repository-level file packing. Rather than simply concatenating files, the IBM team developed a system to arrange files based on semantic dependencies, using directed acyclic graphs derived from analyzing file imports and API calls. This ensures that when the model receives a massive input, the code snippets are ordered logically—much like reading an entire project by following the dependency tree rather than jumping randomly between unrelated source files.
The performance gains on long-context tasks are stark. On benchmarks designed to test understanding across entire codebases, such as Long Code Completion (LCC) and RepoBench-P, the 128K models delivered scores that far exceeded their short-context counterparts.
The most intuitive results come from retrieval tasks like RepoQA, a “Needle-in-a-Haystack” challenge tailored for code. Original Granite models (using 4K context) demonstrated a near-total inability to locate a specific function buried deep within 16,000 tokens of code, often achieving retrieval accuracy of 0%. In contrast, the scaled Granite-8B model achieved a 58% absolute improvement in retrieval accuracy, demonstrating a superior ability to find and utilize relevant information across vast input fields.
Significantly, this context extension was achieved without compromising the model’s established performance on conventional short-context tasks. Evaluations on the HumanEvalPack benchmark showed no noticeable degradation in standard code synthesis, ensuring that the enhanced capabilities come at minimal operational cost.
IBM is releasing all long-context Granite Code models—both the base and instruction-tuned versions—under an Apache 2.0 license, making the advanced 128K capabilities accessible for both research and commercial use, and signaling a commitment to raising the bar for open-source code intelligence.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.