LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs

TL;DR

LCIRC introduces a recurrent compression method for large language models to efficiently handle long-form contexts and query-dependent information, enhancing their ability to process extended sequences without retraining.

Contribution

The paper presents LCIRC, a novel recurrent compression technique that extends LLM context length and incorporates query-dependent relevance without retraining.

Findings

Significantly improves long-form context processing in LLMs

Enhances query relevance in extended contexts

Reduces computational costs for long sequence processing

Abstract

While large language models (LLMs) excel in generating coherent and contextually rich outputs, their capacity to efficiently handle long-form contexts is limited by fixed-length position embeddings. Additionally, the computational cost of processing long sequences increases quadratically, making it challenging to extend context length. To address these challenges, we propose Long-form Context Injection with Recurrent Compression (LCIRC), a method that enables the efficient processing long-form sequences beyond the model's length limit through recurrent compression without retraining the entire model. We further introduce query dependent context modeling, which selectively compresses query-relevant information, ensuring that the model retains the most pertinent content. Our empirical results demonstrate that Query Dependent LCIRC (QD-LCIRC) significantly improves LLM's ability to manage extended contexts, making it well-suited for tasks that require both comprehensive context understanding and query relevance.