Comparing the information content of probabilistic representation spaces

TL;DR

This paper introduces new information-theoretic measures for comparing probabilistic representation spaces, enabling better understanding of learned representations and their information content across models and datasets.

Contribution

It extends classic comparison methods to probabilistic spaces, proposes a lightweight estimation technique, and demonstrates applications in disentanglement, model comparison, and fusion.

Findings

Identified recurring information fragments in latent dimensions of VAEs and InfoGANs.

Revealed consistent information content across models and datasets despite training variability.

Enabled model fusion by leveraging differentiable information measures.

Abstract

Probabilistic representation spaces convey information about a dataset and are shaped by factors such as the training data, network architecture, and loss function. Comparing the information content of such spaces is crucial for understanding the learning process, yet most existing methods assume point-based representations, neglecting the distributional nature of probabilistic spaces. To address this gap, we propose two information-theoretic measures to compare general probabilistic representation spaces by extending classic methods to compare the information content of hard clustering assignments. Additionally, we introduce a lightweight method of estimation that is based on fingerprinting a representation space with a sample of the dataset, designed for scenarios where the communicated information is limited to a few bits. We demonstrate the utility of these measures in three case studies. First, in the context of unsupervised disentanglement, we identify recurring information fragments within individual latent dimensions of VAE and InfoGAN ensembles. Second, we compare the full latent spaces of models and reveal consistent information content across datasets and methods, despite variability during training. Finally, we leverage the differentiability of our measures to perform model fusion, synthesizing the information content of weak learners into a single, coherent representation. Across these applications, the direct comparison of information content offers a natural basis for characterizing the processing of information.