Large Language Model as Token Compressor and Decompressor

TL;DR

This paper introduces a method to adapt large language models into token compressors and decompressors, enabling efficient long-context processing by encoding texts into compact latent codes with minimal performance loss.

Contribution

It presents a self-expressive autoencoding framework using LoRA adapters to create content-adaptive, variable-length token compression for long texts.

Findings

Preserves reconstruction quality on long-context datasets

Reduces memory usage and latency during generation

Supports direct decoding and autoregressive generation in compressed space

Abstract

In this paper, we study whether an off-the-shelf LLM can be adapted into a discrete, variable-length token compressor and decompressor for long-context processing. To this end, we design a self-expressive autoencoding framework that fine-tunes a pretrained LLM with lightweight LoRA adapters to map long texts into compact sequences of learned latent codes, termed Z-tokens, and to decode them back into natural language or task outputs. The resulting representation is content-adaptive: less predictable or information-dense segments can receive more Z-tokens, while redundant regions can be represented more compactly through a budget-aware length regularizer. Our method is evaluated on long-context datasets such as Wikipedia, CNN/DailyMail, HotpotQA, and QuALITY, showing that it preserves reconstruction quality and downstream performance while reducing effective context length, generation-stage memory usage, and end-to-end latency. This simple design supports both direct decoding from compressed contexts and autoregressive generation in the Z-token space, providing a practical interface for efficient long-context inference.