Kinetic variable-sample methods for stochastic optimization problems

TL;DR

This paper introduces a novel kinetic optimization algorithm that combines particle methods and variable-sample strategies to efficiently solve stochastic problems with convergence guarantees and improved computational performance.

Contribution

It develops a new kinetic-based algorithm integrating instantaneous collisions and variable sampling, with proven convergence and demonstrated efficiency improvements.

Findings

Proved convergence of the new kinetic method under parameter constraints.

Established a connection to existing consensus-based stochastic optimization methods.

Validated the approach through numerical experiments showing enhanced efficiency.

Abstract

We discuss kinetic-based particle optimization methods and variable-sample strategies for problems where the cost function represents the expected value of a random mapping. Kinetic-based optimization methods rely on a consensus mechanism targeting the global minimizer, and they exploit tools of kinetic theory to establish a rigorous framework for proving convergence to that minimizer. Variable-sample strategies replace the expected value by an approximation at each iteration of the optimization algorithm. We combine these approaches and introduce a novel algorithm based on instantaneous collisions governed by a linear Boltzmann-type equation. After proving the convergence of the resulting kinetic method under appropriate parameter constraints, we establish a connection to a recently introduced consensus-based method for solving the random problem in a suitable scaling. Finally, we showcase its enhanced computational efficiency compared to the aforementioned algorithm and validate the consistency of the proposed modeling approaches through several numerical experiments.