Neural Normalized Compression Distance and the Disconnect Between Compression and Classification

TL;DR

This paper investigates the relationship between compression and classification by developing a Neural Normalized Compression Distance using large language models, revealing that compression rate alone does not predict classification accuracy.

Contribution

It introduces Neural NCD leveraging LLMs as lossless compressors and demonstrates the disconnect between compression and classification performance.

Findings

Classification accuracy is not solely determined by compression rate.

Neural compression behaviors differ from traditional algorithms.

Current understanding of neural compression and classification is incomplete.

Abstract

It is generally well understood that predictive classification and compression are intrinsically related concepts in information theory. Indeed, many deep learning methods are explained as learning a kind of compression, and that better compression leads to better performance. We interrogate this hypothesis via the Normalized Compression Distance (NCD), which explicitly relies on compression as the means of measuring similarity between sequences and thus enables nearest-neighbor classification. By turning popular large language models (LLMs) into lossless compressors, we develop a Neural NCD and compare LLMs to classic general-purpose algorithms like gzip. In doing so, we find that classification accuracy is not predictable by compression rate alone, among other empirical aberrations not predicted by current understanding. Our results imply that our intuition on what it means for a neural network to ``compress'' and what is needed for effective classification are not yet well understood.