Knowledge as a Breaking of Ergodicity

TL;DR

This paper models the training of generative models as a breaking of ergodicity, where multiple minima in the free energy surface correspond to distinct learned states, affecting the model's ability to generalize and retrieve patterns.

Contribution

It introduces a thermodynamic framework for understanding generative model training as ergodicity breaking, highlighting the emergence of multiple minima and their implications.

Findings

Training creates multiple free energy minima representing learned states.

Ergodicity breaking limits access to underrepresented patterns.

Using multiple models per minimum can improve performance.

Abstract

We construct a thermodynamic potential that can guide training of a generative model defined on a set of binary degrees of freedom. We argue that upon reduction in description, so as to make the generative model computationally-manageable, the potential develops multiple minima. This is mirrored by the emergence of multiple minima in the free energy proper of the generative model itself. The variety of training samples that employ N binary degrees of freedom is ordinarily much lower than the size 2^N of the full phase space. The non-represented configurations, we argue, should be thought of as comprising a high-temperature phase separated by an extensive energy gap from the configurations composing the training set. Thus, training amounts to sampling a free energy surface in the form of a library of distinct bound states, each of which breaks ergodicity. The ergodicity breaking prevents escape into the near continuum of states comprising the high-temperature phase; thus it is necessary for proper functionality. It may however have the side effect of limiting access to patterns that were underrepresented in the training set. At the same time, the ergodicity breaking within the library complicates both learning and retrieval. As a remedy, one may concurrently employ multiple generative models -- up to one model per free energy minimum.