Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

TL;DR

This paper evaluates language models for lossless audio compression across various domains and bit depths, introducing Trilobyte for efficient 24-bit audio tokenization, and compares performance with existing codecs.

Contribution

It introduces Trilobyte, a byte-level tokenization schema, enabling tractable 24-bit lossless compression with language models, and benchmarks their performance on full-fidelity audio.

Findings

LMs outperform FLAC at 8-bit and 16-bit

Trilobyte enables 24-bit lossless compression

Compression gains diminish beyond 8-bit

Abstract

Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs. We benchmark LM-based compression on full-fidelity audio across diverse domains (music, speech, bioacoustics), sampling rates (16kHz-48kHz), and bit depths (8, 16, 24-bit). Standard sample-level tokenization becomes intractable at higher bit depths due to vocabulary size (65K for 16-bit; 16.7M for 24-bit). We propose Trilobyte, a byte-level tokenization schema for full resolution audio, improving vocabulary scaling from $O(2^{b})$ to $O(1)$ and enabling the first tractable 24-bit LM-based lossless compression. While LMs consistently outperform FLAC and yield state-of-the-art compression at 8-bit and 16-bit, we observe that compression gains become more modest as bit depth increases beyond 8-bit.