A consensus-based optimization method for nonsmooth nonconvex programs with approximated gradient descent scheme

TL;DR

This paper introduces a novel consensus-based optimization algorithm that combines discrete consensus, gradient descent, and function value evaluations to efficiently find global minima in nonsmooth nonconvex problems, with proven convergence properties.

Contribution

The paper develops a new CBO algorithm with an integrated gradient descent scheme using only function values, providing theoretical convergence guarantees without mean-field assumptions.

Findings

Proven exponential convergence to a global consensus point.

Error estimate shows convergence to the global minimum as parameter increases.

Experimental results demonstrate improved efficiency on benchmark problems and neural network training.

Abstract

In this paper, we are interested in finding the global minimizer of a nonsmooth nonconvex unconstrained optimization problem. By combining the discrete consensus-based optimization (CBO) algorithm and the gradient descent method, we develop a novel CBO algorithm with an extra gradient descent scheme evaluated by the forward-difference technique on the function values, where only the objective function values are used in the proposed algorithm. First, we prove that the proposed algorithm can exhibit global consensus in an exponential rate in two senses and possess a unique global consensus point. Second, we evaluate the error estimate between the objective function value on the global consensus point and its global minimum. In particular, as the parameter $\beta$ tends to $\infty$, the error converges to zero and the convergence rate is $\mathcal{O}\left(\frac{\log\beta}{\beta}\right)$. Third, under some suitable assumptions on the objective function, we provide the number of iterations required for the mean square error in expectation to reach the desired accuracy. It is worth underlining that the theoretical analysis in this paper does not use the mean-field limit. Finally, we illustrate the improved efficiency and promising performance of our novel CBO method through some experiments on several nonconvex benchmark problems and the application to train deep neural networks.