Mean-field inference methods for neural networks

TL;DR

This paper reviews classical and recent mean-field methods used for inference in neural networks, connecting statistical physics approaches with machine learning techniques to enhance understanding of neural network behavior.

Contribution

It provides a comprehensive overview of mean-field approximation strategies, including derivations, equivalences, and recent advances relevant for neural network inference.

Findings

Clarifies the connections between different mean-field methods.

Highlights recent progress in applying statistical physics to neural networks.

Provides references for ongoing research directions.

Abstract

Machine learning algorithms relying on deep neural networks recently allowed a great leap forward in artificial intelligence. Despite the popularity of their applications, the efficiency of these algorithms remains largely unexplained from a theoretical point of view. The mathematical description of learning problems involves very large collections of interacting random variables, difficult to handle analytically as well as numerically. This complexity is precisely the object of study of statistical physics. Its mission, originally pointed towards natural systems, is to understand how macroscopic behaviors arise from microscopic laws. Mean-field methods are one type of approximation strategy developed in this view. We review a selection of classical mean-field methods and recent progress relevant for inference in neural networks. In particular, we remind the principles of derivations of high-temperature expansions, the replica method and message passing algorithms, highlighting their equivalences and complementarities. We also provide references for past and current directions of research on neural networks relying on mean-field methods.