DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search

TL;DR

This paper introduces DCIS, a divide-and-conquer algorithm for efficiently determining optimal scaling factors to extend LLMs' context length, reducing fine-tuning costs and improving performance at longer contexts.

Contribution

The paper proposes a novel DCIS algorithm that strategically searches for scaling factors, enabling effective context length extension with less fine-tuning and higher efficiency than existing methods.

Findings

DCIS doubles search efficiency compared to other methods.

The identified scaling factors improve performance at extended lengths.

Models can generalize to longer contexts without additional fine-tuning.

Abstract

Large language models (LLMs) based on the Transformer architecture usually have their context length limited due to the high training cost. Recent advancements extend the context window by adjusting the scaling factors of RoPE and fine-tuning. However, suboptimal initialization of these factors results in increased fine-tuning costs and reduced performance at target length. To address these challenges, we propose a novel RoPE-based fine-tuning framework that diverges from conventional scaling factors search. Specifically, we present a \textbf{D}ivide-and-\textbf{C}onquer \textbf{I}ncremental \textbf{S}earch (DCIS) algorithm that strategically determines the better scaling factors. Further fine-tuning with the identified scaling factors effectively extends the context window of LLMs. Empirical results demonstrate that our methodology not only mitigates performance decay at extended target lengths but also allows the model to fine-tune on short contexts and generalize to long contexts, thereby reducing the cost of fine-tuning. The scaling factors obtained through DCIS can even perform effectively without fine-tuning. Further analysis of the search space reveals that DCIS achieves twice the search efficiency compared to other methods. We also examine the impact of the non-strictly increasing scaling factors utilized in DCIS and evaluate the general capabilities of LLMs across various context lengths.