Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity

TL;DR

This paper investigates the maximum lossless compression of token sequences into single vectors, revealing that ratios up to 1500x are theoretically possible, far exceeding current methods, and that limits are set by uncertainty reduction rather than input length.

Contribution

It introduces a per-sample optimization approach to explore the theoretical limits of embedding space capacity, demonstrating much higher compression ratios than existing techniques.

Findings

Compression ratios up to x1500 are achievable.

Limits are dictated by sequence uncertainty, not input length.

Significant potential for optimizing model design to utilize embedding capacity.

Abstract

A range of recent works addresses the problem of compression of sequence of tokens into a shorter sequence of real-valued vectors to be used as inputs instead of token embeddings or key-value cache. These approaches are focused on reduction of the amount of compute in existing language models rather than minimization of number of bits needed to store text. Despite relying on powerful models as encoders, the maximum attainable lossless compression ratio is typically not higher than x10. This fact is highly intriguing because, in theory, the maximum information capacity of large real-valued vectors is far beyond the presented rates even for 16-bit precision and a modest vector size. In this work, we explore the limits of compression by replacing the encoder with a per-sample optimization procedure. We show that vectors with compression ratios up to x1500 exist, which highlights two orders of magnitude gap between existing and practically attainable solutions. Furthermore, we empirically show that the compression limits are determined not by the length of the input but by the amount of uncertainty to be reduced, namely, the cross-entropy loss on this sequence without any conditioning. The obtained limits highlight the substantial gap between the theoretical capacity of input embeddings and their practical utilization, suggesting significant room for optimization in model design.