Mean Field Optimization Problem Regularized by Fisher Information

TL;DR

This paper introduces a Fisher Information regularized mean field optimization framework, connecting it to Schrödinger dynamics, and demonstrates exponential convergence of distributions with a new sampling method.

Contribution

It establishes a novel link between Fisher regularization in mean field optimization and Schrödinger dynamics, including convergence analysis and a Monte Carlo sampling approach.

Findings

Exponential convergence of marginal distributions to the optimizer.

Mean field Schrödinger dynamics as a gradient flow on probability measures.

Proposed Monte Carlo method for sampling the distributions.

Abstract

Recently there is a rising interest in the research of mean field optimization, in particular because of its role in analyzing the training of neural networks. In this paper by adding the Fisher Information as the regularizer, we relate the regularized mean field optimization problem to a so-called mean field Schrodinger dynamics. We develop an energy-dissipation method to show that the marginal distributions of the mean field Schrodinger dynamics converge exponentially quickly towards the unique minimizer of the regularized optimization problem. Remarkably, the mean field Schrodinger dynamics is proved to be a gradient flow on the probability measure space with respect to the relative entropy. Finally we propose a Monte Carlo method to sample the marginal distributions of the mean field Schrodinger dynamics.