Scaling Granite Code Models to 128K Context

TL;DR

This paper presents a method to scale Granite code models to support 128K token contexts, enabling better long-context understanding without sacrificing short-context performance.

Contribution

The authors introduce a lightweight pretraining approach and release long-context Granite models, significantly extending context length capabilities for code models.

Findings

Long-context models outperform short-context models on long tasks.

No performance loss on standard code benchmarks.

Models are released under an open license.

Abstract

This paper introduces long-context Granite code models that support effective context windows of up to 128K tokens. Our solution for scaling context length of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight continual pretraining by gradually increasing its RoPE base frequency with repository-level file packing and length-upsampled long-context data. Additionally, we also release instruction-tuned models with long-context support which are derived by further finetuning the long context base models on a mix of permissively licensed short and long-context instruction-response pairs. While comparing to the original short-context Granite code models, our long-context models achieve significant improvements on long-context tasks without any noticeable performance degradation on regular code completion benchmarks (e.g., HumanEval). We release all our long-context Granite code models under an Apache 2.0 license for both research and commercial use.