Language Modeling Is Compression

TL;DR

This paper explores the idea that large language models can be viewed as powerful compressors, providing new insights into their capabilities and enabling the use of general-purpose compressors for generative modeling.

Contribution

It demonstrates that large language models are effective predictors and compressors, offering a novel perspective that links compression with prediction and enables new applications.

Findings

Chinchilla 70B compresses ImageNet and LibriSpeech better than domain-specific compressors.

Large language models exhibit strong general-purpose prediction and compression abilities.

The prediction-compression equivalence allows using standard compressors for generative modeling.

Abstract

It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Finally, we show that the prediction-compression equivalence allows us to use any compressor (like gzip) to build a conditional generative model.