Mean-Field Langevin Dynamics: Exponential Convergence and Annealing

TL;DR

This paper proves exponential convergence of Mean-Field Langevin dynamics under certain conditions and analyzes its behavior with decreasing noise, providing theoretical insights into its efficiency for convex measure minimization.

Contribution

It establishes exponential convergence rates for Mean-Field Langevin dynamics assuming Log-Sobolev inequalities and analyzes the annealed dynamics with decreasing noise.

Findings

Proves exponential convergence under Log-Sobolev inequalities.

Shows convergence of annealed dynamics with logarithmic noise decay.

Applicable to neural network risk minimization.

Abstract

Noisy particle gradient descent (NPGD) is an algorithm to minimize convex functions over the space of measures that include an entropy term. In the many-particle limit, this algorithm is described by a Mean-Field Langevin dynamics - a generalization of the Langevin dynamics with a non-linear drift - which is our main object of study. Previous work have shown its convergence to the unique minimizer via non-quantitative arguments. We prove that this dynamics converges at an exponential rate, under the assumption that a certain family of Log-Sobolev inequalities holds. This assumption holds for instance for the minimization of the risk of certain two-layer neural networks, where NPGD is equivalent to standard noisy gradient descent. We also study the annealed dynamics, and show that for a noise decaying at a logarithmic rate, the dynamics converges in value to the global minimizer of the unregularized objective function.