Phase transitions in optimal unsupervised learning

TL;DR

This paper analyzes the optimal performance and phase transitions in an unsupervised learning problem involving high-dimensional data with symmetry-breaking features, revealing conditions for first-order transitions and metastable states.

Contribution

It provides a theoretical analysis of the phase transitions and metastable states in optimal unsupervised learning of symmetry axes in high-dimensional data.

Findings

Identification of first-order phase transitions in learning curves

Existence of metastable high-performance states near transitions

Conditions under which optimal solutions are not learnable in Bayesian setting

Abstract

We determine the optimal performance of learning the orientation of the symmetry axis of a set of P = alpha N points that are uniformly distributed in all the directions but one on the N-dimensional sphere. The components along the symmetry breaking direction, of unitary vector B, are sampled from a mixture of two gaussians of variable separation and width. The typical optimal performance is measured through the overlap Ropt=B.J* where J* is the optimal guess of the symmetry breaking direction. Within this general scenario, the learning curves Ropt(alpha) may present first order transitions if the clusters are narrow enough. Close to these transitions, high performance states can be obtained through the minimization of the corresponding optimal potential, although these solutions are metastable, and therefore not learnable, within the usual bayesian scenario.