# The peculiar statistical mechanics of Optimal Learning Machines

## TL;DR

This paper explores the statistical mechanics of Optimal Learning Machines, revealing their unique position at a phase transition boundary characterized by a stretched exponential energy distribution, with implications for learnability and predictability.

## Contribution

It introduces a theoretical framework linking OLM properties to a specific energy distribution exponent, explaining their independence from environment size and phase transition behavior.

## Key findings

- OLM correspond to a critical case with exponential energy distribution.
- Systems with different energy distribution exponents exhibit distinct phase transition types.
- OLM's behavior remains stable regardless of environment size, indicating efficient representation.

## Abstract

Optimal Learning Machines (OLM) are systems that extract maximally informative representation of the environment they are in contact with, or of the data they are presented. It has recently been suggested that these systems are characterised by an exponential distribution of energy levels. In order to understand the peculiar properties of OLM within a broader framework, I consider an ensemble of optimisation problems over functions of many variables, part of which describe a sub-system and the rest account for its interaction with a random environment. The number of states of the sub-system with a given value of the objective function obeys a stretched exponential distribution, with exponent $\gamma$, and the interaction part is drawn at random from the same distribution, independently for each configuration of the whole system. Systems with $\gamma=1$ then correspond to OLM, and we find that they sit at the boundary between two regions with markedly different properties. For all $\gamma>0$ the system exhibits a freezing phase transition. The transition is discontinuous for $\gamma<1$ and it is continuous for $\gamma>1$. The region $\gamma>1$ corresponds to learnable energy landscapes and the behaviour of the sub-system becomes predictable as the size of the environment exceeds a critical threshold. For $\gamma<1$, instead, the energy landscape is unlearnable and the behaviour of the system becomes more and more unpredictable as the size of the environment increases. Sub-systems with $\gamma=1$ (OLM) feature a behaviour which is independent of the relative size of the environment. This is consistent with the expectation that efficient representations should be largely independent of the level of detail of the description of the environment.