Recurrent Context Compression: Efficiently Expanding the Context Window of LLM

TL;DR

This paper introduces Recurrent Context Compression (RCC), a method to significantly extend the context window of LLMs efficiently, enabling better long-text understanding and retrieval with minimal resource increase.

Contribution

The paper proposes RCC, a novel technique for expanding LLM context windows efficiently, along with an instruction reconstruction method to maintain response quality during compression.

Findings

Achieved up to 32x compression with high BLEU4 scores

Nearly 100% accuracy on 1M sequence passkey retrieval

Competitive long-text QA performance with resource savings

Abstract

To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100\% accuracy on a passkey retrieval task with a sequence length of 1M. Finally, our method demonstrated competitive performance in long-text question-answering tasks compared to non-compressed methods, while significantly saving storage resources in long-text inference tasks. Our code, models, and demo are available at https://github.com/WUHU-G/RCC_Transformer