Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law

TL;DR

This paper introduces a variant of consensus-based optimization that incorporates memory effects and gradient information, providing rigorous proof of global convergence in mean-field law and demonstrating improved performance in machine learning and compressed sensing.

Contribution

It presents a new CBO variant with memory and gradient features, proving its global convergence and showing practical advantages over traditional methods.

Findings

Proves global convergence of the new CBO variant in mean-field law.

Demonstrates improved performance in machine learning tasks.

Shows superiority in compressed sensing applications.

Abstract

In this paper we study consensus-based optimization (CBO), a versatile, flexible and customizable optimization method suitable for performing nonconvex and nonsmooth global optimizations in high dimensions. CBO is a multi-particle metaheuristic, which is effective in various applications and at the same time amenable to theoretical analysis thanks to its minimalistic design. The underlying dynamics, however, is flexible enough to incorporate different mechanisms widely used in evolutionary computation and machine learning, as we show by analyzing a variant of CBO which makes use of memory effects and gradient information. We rigorously prove that this dynamics converges to a global minimizer of the objective function in mean-field law for a vast class of functions under minimal assumptions on the initialization of the method. The proof in particular reveals how to leverage further, in some applications advantageous, forces in the dynamics without loosing provable global convergence. To demonstrate the benefit of the herein investigated memory effects and gradient information in certain applications, we present numerical evidence for the superiority of this CBO variant in applications such as machine learning and compressed sensing, which en passant widen the scope of applications of CBO.