Relevance in the Renormalization Group and in Information Theory

TL;DR

This paper establishes a theoretical link between the information-theoretic concept of relevance in the Information Bottleneck method and the physical notion of relevance in the Renormalization Group, providing insights into interpretability in physics-inspired machine learning.

Contribution

It demonstrates an analytical and numerical connection between IB relevance and RG relevance, bridging information theory and field theory in physical systems.

Findings

IB relevance corresponds to operators with lowest scaling dimensions in field theory

Numerical confirmation of the analytical predictions

Dependence of IB solutions on physical symmetries analyzed

Abstract

The analysis of complex physical systems hinges on the ability to extract the relevant degrees of freedom from among the many others. Though much hope is placed in machine learning, it also brings challenges, chief of which is interpretability. It is often unclear what relation, if any, the architecture- and training-dependent learned "relevant" features bear to standard objects of physical theory. Here we report on theoretical results which may help to systematically address this issue: we establish equivalence between the information-theoretic notion of relevance defined in the Information Bottleneck (IB) formalism of compression theory, and the field-theoretic relevance of the Renormalization Group. We show analytically that for statistical physical systems described by a field theory the "relevant" degrees of freedom found using IB compression indeed correspond to operators with the lowest scaling dimensions. We confirm our field theoretic predictions numerically. We study dependence of the IB solutions on the physical symmetries of the data. Our findings provide a dictionary connecting two distinct theoretical toolboxes, and an example of constructively incorporating physical interpretability in applications of deep learning in physics.