Emergence of Structure in Ensembles of Random Neural Networks

TL;DR

This paper develops a theoretical model to understand how ensembles of random neural classifiers can self-organize into optimal collective behaviors, revealing universal properties and potential implications for machine learning systems.

Contribution

It introduces a Gibbs measure-based framework for analyzing collective behavior in random classifier ensembles, demonstrating universality and optimality conditions.

Findings

Optimal classification occurs at a specific finite temperature.

The optimal temperature is independent of the teacher classifier and ensemble size.

Experimental validation on MNIST supports the theoretical predictions.

Abstract

Randomness is ubiquitous in many applications across data science and machine learning. Remarkably, systems composed of random components often display emergent global behaviors that appear deterministic, manifesting a transition from microscopic disorder to macroscopic organization. In this work, we introduce a theoretical model for studying the emergence of collective behaviors in ensembles of random classifiers. We argue that, if the ensemble is weighted through the Gibbs measure defined by adopting the classification loss as an energy, then there exists a finite temperature parameter for the distribution such that the classification is optimal, with respect to the loss (or the energy). Interestingly, for the case in which samples are generated by a Gaussian distribution and labels are constructed by employing a teacher perceptron, we analytically prove and numerically confirm that such optimal temperature does not depend neither on the teacher classifier (which is, by construction of the learning problem, unknown), nor on the number of random classifiers, highlighting the universal nature of the observed behavior. Experiments on the MNIST dataset underline the relevance of this phenomenon in high-quality, noiseless, datasets. Finally, a physical analogy allows us to shed light on the self-organizing nature of the studied phenomenon.