Mean-field particle swarm optimization

TL;DR

This paper analyzes mean-field particle swarm optimization (PSO) methods for high-dimensional, non-convex optimization problems, providing theoretical insights and convergence results through mean-field and hydrodynamic limits, supported by numerical experiments.

Contribution

It introduces a mean-field framework for PSO, deriving Vlasov-Fokker-Planck equations and analyzing the zero-inertia limit, extending the understanding of PSO's theoretical foundations.

Findings

Mean-field limit of PSO derived using Vlasov-Fokker-Planck equations

Zero-inertia limit leads to hydrodynamic equations generalizing CBO methods

Convergence towards the global minimum established with numerical validation

Abstract

In this work we survey some recent results on the global minimization of a non-convex and possibly non-smooth high dimensional objective function by means of particle based gradient-free methods. Such problems arise in many situations of contemporary interest in machine learning and signal processing. After a brief overview of metaheuristic methods based on particle swarm optimization (PSO), we introduce a continuous formulation via second-order systems of stochastic differential equations that generalize PSO methods and provide the basis for their theoretical analysis. Subsequently, we will show how through the use of mean-field techniques it is possible to derive in the limit of large particles number the corresponding mean-field PSO description based on Vlasov-Fokker-Planck type equations. Finally, in the zero inertia limit, we will analyze the corresponding macroscopic hydrodynamic equations, showing that they generalize the recently introduced consensus-based optimization (CBO) methods by including memory effects. Rigorous results concerning the mean-field limit, the zero-inertia limit, and the convergence of the mean-field PSO method towards the global minimum are provided along with a suite of numerical examples.