Maximal Relevance and Optimal Learning Machines

TL;DR

This paper introduces the concept of maximal relevance in learning machines, showing that models with maximal relevance extract the most informative representations and perform optimally on various tasks, including neural network training.

Contribution

It defines and explores the principle of maximal relevance in learning models, linking it to optimal information extraction and demonstrating its effects across different models and tasks.

Findings

OLMs have inhomogeneous coupling distributions.

Likelihood maximization aligns with maximal relevance.

Training RBMs on MNIST approaches maximal relevance.

Abstract

We show that the mutual information between the representation of a learning machine and the hidden features that it extracts from data is bounded from below by the relevance, which is the entropy of the model's energy distribution. Models with maximal relevance -- that we call Optimal Learning Machines (OLM) -- are hence expected to extract maximally informative representations. We explore this principle in a range of models. For fully connected Ising models and we show that {\em i)} OLM are characterised by inhomogeneous distributions of couplings, and that {\em ii)} their learning performance is affected by sub-extensive features that are elusive to a thermodynamic treatment. On specific learning tasks, we find that likelihood maximisation is achieved by models with maximal relevance. Training of Restricted Boltzmann Machines on the MNIST benchmark shows that learning is associated with a broadening of the spectrum of energy levels and that the internal representation of the hidden layer approaches the maximal relevance that can be achieved in a finite dataset. Finally, we discuss a Gaussian learning machine that clarifies that learning hidden features is conceptually different from parameter estimation.