In-Context Former: Lightning-fast Compressing Context for Large Language Model

TL;DR

In-Context Former (IC-Former) is a novel method that compresses large language model contexts efficiently by leveraging cross-attention and digest tokens, significantly reducing inference costs and enabling real-time applications.

Contribution

IC-Former introduces a cross-attention based compression method that operates independently of target LLMs, achieving linear time complexity and substantial speedups over existing quadratic methods.

Findings

Requires only 1/32 of the baseline FLOPs during compression

Achieves 68 to 112 times faster processing speed

Maintains over 90% of baseline performance on evaluation metrics

Abstract

With the rising popularity of Transformer-based large language models (LLMs), reducing their high inference costs has become a significant research focus. One effective approach is to compress the long input contexts. Existing methods typically leverage the self-attention mechanism of the LLM itself for context compression. While these methods have achieved notable results, the compression process still involves quadratic time complexity, which limits their applicability. To mitigate this limitation, we propose the In-Context Former (IC-Former). Unlike previous methods, IC-Former does not depend on the target LLMs. Instead, it leverages the cross-attention mechanism and a small number of learnable digest tokens to directly condense information from the contextual word embeddings. This approach significantly reduces inference time, which achieves linear growth in time complexity within the compression range. Experimental results indicate that our method requires only 1/32 of the floating-point operations of the baseline during compression and improves processing speed by 68 to 112 times while achieving over 90% of the baseline performance on evaluation metrics. Overall, our model effectively reduces compression costs and makes real-time compression scenarios feasible.