Surveying the space of descriptions of a composite system with machine learning

TL;DR

This paper introduces a machine learning framework to explore the continuous space of descriptions of complex systems, enabling detailed analysis of their organizational structure through information theoretic measures.

Contribution

It proposes a novel approach combining machine learning with information theory to analyze the organization of complex systems via their descriptions.

Findings

Identified extremal descriptions revealing system-wide variation

Applied framework to spin systems, sudoku, and language sequences

Demonstrated detailed insights into system organization

Abstract

Multivariate information theory provides a general and principled framework for understanding how the components of a complex system are connected. Existing analyses are coarse in nature -- built up from characterizations of discrete subsystems -- and can be computationally prohibitive. In this work, we propose to study the continuous space of possible descriptions of a composite system as a window into its organizational structure. A description consists of specific information conveyed about each of the components, and the space of possible descriptions is equivalent to the space of lossy compression schemes of the components. We introduce a machine learning framework to optimize descriptions that extremize key information theoretic quantities used to characterize organization, such as total correlation and O-information. Through case studies on spin systems, sudoku boards, and letter sequences from natural language, we identify extremal descriptions that reveal how system-wide variation emerges from individual components. By integrating machine learning into a fine-grained information theoretic analysis of composite random variables, our framework opens a new avenues for probing the structure of real-world complex systems.