Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data

TL;DR

This study demonstrates that small pre-trained transformer models can outperform traditional compression algorithms on byte-level multimodal data, highlighting their potential as effective data compressors.

Contribution

It provides a large-scale empirical analysis of pre-trained transformers for compression, identifying optimal model sizes and training strategies across multiple data modalities.

Findings

Small models outperform standard compression algorithms.

Multimodal training improves performance on multiple data types.

Transferability to unseen modalities is limited.

Abstract

Foundation models are strong data compressors, but when accounting for their parameter size, their compression ratios are inferior to standard compression algorithms. Naively reducing the parameter count does not necessarily help as it deteriorates predictions and, accordingly, compression. We conduct a large-scale empirical study to find a sweet spot where pre-trained vanilla transformers can achieve competitive compression ratios. To this end, we train models on 165GB of raw byte sequences of either text, image, or audio data (and all possible combinations of the three) and then compress 1GB of out-of-distribution (OOD) data from each modality. We find that relatively small models (millions of parameters) can outperform standard general-purpose compression algorithms (gzip, LZMA2) and even domain-specific compressors (PNG, JPEG-XL, FLAC) $\unicode{x2013}$ even when accounting for parameter size. We achieve, e.g., the lowest compression ratio of 0.49 on OOD audio data (vs. 0.54 for FLAC). We conduct extensive ablations and hyperparameter sweeps to study the impact of model- and dataset scale, and we investigate the effect of unimodal versus multimodal training. We find that even small models can be trained to perform well on multiple modalities, but unlike large-scale foundation models, transfer to unseen modalities is generally weak.