Information-Ordered Bottlenecks for Adaptive Semantic Compression

TL;DR

The paper introduces the information-ordered bottleneck (IOB), a neural layer that adaptively compresses data into semantically ordered latent variables, achieving near-optimal compression and enabling meaningful data analysis without retraining.

Contribution

The paper proposes IOB, a novel neural layer that orders latent variables by importance, unifies previous approaches, and provides a new theory for intrinsic dimensionality estimation.

Findings

IOB achieves near-optimal compression across architectures.

IOB accurately estimates intrinsic data dimensionality.

IOB enables exploratory analysis of complex datasets.

Abstract

We present the information-ordered bottleneck (IOB), a neural layer designed to adaptively compress data into latent variables ordered by likelihood maximization. Without retraining, IOB nodes can be truncated at any bottleneck width, capturing the most crucial information in the first latent variables. Unifying several previous approaches, we show that IOBs achieve near-optimal compression for a given encoding architecture and can assign ordering to latent signals in a manner that is semantically meaningful. IOBs demonstrate a remarkable ability to compress embeddings of image and text data, leveraging the performance of SOTA architectures such as CNNs, transformers, and diffusion models. Moreover, we introduce a novel theory for estimating global intrinsic dimensionality with IOBs and show that they recover SOTA dimensionality estimates for complex synthetic data. Furthermore, we showcase the utility of these models for exploratory analysis through applications on heterogeneous datasets, enabling computer-aided discovery of dataset complexity.