Lossless Token Sequence Compression via Meta-Tokens

TL;DR

This paper presents a lossless token sequence compression method for LLMs that reduces input length by around 20-27%, significantly decreasing computation without losing semantic information, outperforming lossy methods in preserving semantics.

Contribution

The paper introduces a task-agnostic, lossless compression technique similar to LZ77 for transformer-based LLMs, reducing input size without semantic loss.

Findings

Reduces token sequence length by 27% and 18% on two tasks.

Decreases encoding computation by 47% and 33%.

Maintains performance close to uncompressed input.

Abstract

Existing work on prompt compression for Large Language Models (LLM) focuses on lossy methods that try to maximize the retention of semantic information that is relevant to downstream tasks while significantly reducing the sequence length. In this paper, we introduce a task-agnostic lossless compression technique similar to LZ77 that makes it possible to reduce the input token sequence length on average by 27\% and 18\% for the two evaluation tasks explored here. Given that we use transformer-based LLMs, this equates to 47\% and 33\% less encoding computation, respectively, due to the quadratic nature of attention. The token sequence transformation is trivial to reverse and highlights that no semantic information is lost in the process. We evaluate our proposed approach on two tasks that require strict preservation of semantics/syntax and demonstrate that existing lossy compression methods perform poorly in this setting. We find that our lossless compression technique produces only a small gap in performance compared to using the uncompressed input and posit that larger models and an expanded computing budget would likely erase the gap entirely.