Sentence-Anchored Gist Compression for Long-Context LLMs

TL;DR

This paper introduces a learned compression token method for large language models that significantly reduces context size by up to 8x with minimal performance loss, enabling more efficient long-sequence processing.

Contribution

It presents a novel fine-tuning approach for LLMs to compress context using learned tokens, achieving higher compression ratios than existing methods.

Findings

Compression factors of 2x to 8x without performance loss

Comparable results to alternative methods on benchmarks

Higher compression ratios achieved with minimal accuracy impact

Abstract

This work investigates context compression for Large Language Models (LLMs) using learned compression tokens to reduce the memory and computational demands of processing long sequences. We demonstrate that pre-trained LLMs can be fine-tuned to compress their context by factors of 2x to 8x without significant performance degradation, as evaluated on both short-context and long-context benchmarks. Furthermore, in experiments on a 3-billion-parameter LLaMA model, our method achieves results on par with alternative compression techniques while attaining higher compression ratios.