Statistical physics of unsupervised learning with prior knowledge in neural networks

TL;DR

This paper develops a statistical physics model to understand how prior knowledge influences unsupervised learning in neural networks, revealing phase transitions and symmetry-breaking phenomena driven by sensory inputs and priors.

Contribution

It introduces a quantitative physics-based framework for analyzing the role of prior knowledge in unsupervised learning, highlighting phase transitions and symmetry effects.

Findings

Prior reduces the data needed for symmetry breaking.

Prior merges permutation symmetry breaking phases.

Sensory inputs induce continuous phase transitions.

Abstract

Integrating sensory inputs with prior beliefs from past experiences in unsupervised learning is a common and fundamental characteristic of brain or artificial neural computation. However, a quantitative role of prior knowledge in unsupervised learning remains unclear, prohibiting a scientific understanding of unsupervised learning. Here, we propose a statistical physics model of unsupervised learning with prior knowledge, revealing that the sensory inputs drive a series of continuous phase transitions related to spontaneous intrinsic-symmetry breaking. The intrinsic symmetry includes both reverse symmetry and permutation symmetry, commonly observed in most artificial neural networks. Compared to the prior-free scenario, the prior reduces more strongly the minimal data size triggering the reverse symmetry breaking transition, and moreover, the prior merges, rather than separates, permutation symmetry breaking phases. We claim that the prior can be learned from data samples, which in physics corresponds to a two-parameter Nishimori constraint. This work thus reveals mechanisms about the influence of the prior on unsupervised learning.