An Informational Parsimony Perspective on Probabilistic Symmetries

TL;DR

This paper introduces a group-theoretic framework for extracting probabilistic symmetries through compression, generalizing the Information Bottleneck, and demonstrates its ability to recover nested symmetries at different resolutions.

Contribution

It formalizes the extraction of probabilistic symmetries using a novel compression-based approach extending the Information Bottleneck framework.

Findings

Successfully recovers nested equivariances at different compression levels

Defines soft symmetries based on divergence preservation and compression

Demonstrates the emergence of new symmetries at bifurcation points

Abstract

Extraction of structure, in particular of group symmetries, is increasingly crucial to understanding and building intelligent models. In particular, some information-theoretic models of parsimonious learning have been argued to induce invariance extraction. Here, we formalise these arguments from a group-theoretic perspective. We then extend them to the study of more general probabilistic symmetries, through compressions preserving geometric measures of complexity. More precisely, our framework implements a trade-off between compression and preservation of the divergence from a given hierarchical model, yielding a novel generalisation of the Information Bottleneck framework. Through appropriate choices of hierarchical models, we fully characterise (in the discrete and full support case) channel invariance, channel equivariance and distribution invariance under permutation. Allowing imperfect divergence preservation then leads to principled definitions of "soft symmetries", where the "coarseness" corresponds to the degree of compression of the system. In simple synthetic experiments, we demonstrate that our method successively recovers, at increasingly compressed "resolutions", nested but increasingly perturbed equivariances, where new equivariances emerge at bifurcation points of the trade-off parameter. Our framework suggests a new path for the extraction of generalised probabilistic symmetries.