Swarm-Based Inertial Methods for Optimization

TL;DR

This paper introduces a new class of swarm-based inertial optimization methods derived from dissipative dynamical systems, providing theoretical convergence guarantees and demonstrating superior performance on convex and nonconvex problems.

Contribution

The paper develops a systematic framework for swarm-based inertial methods using energy dissipation principles, including novel damping mechanisms and structure-preserving discretizations.

Findings

Proves energy dissipation law and convergence rate bounds for the proposed methods.

Constructs discretizations that retain energy decay and convergence properties.

Demonstrates faster convergence and higher success rates on benchmark problems.

Abstract

We introduce a new class of swarm-based inertial methods (SBIMs) for global minimization, formulated as coupled dissipative inertial dynamical systems derived from the generalized Onsager principle. The proposed framework identifies the friction operator and the scaling of the potential energy, namely the objective function to be minimized, as the key ingredients governing relaxation dynamics over the energy landscape. Within this framework, we propose a new underdamped inertial dynamics whose damping mechanisms incorporate both gradient and Hessian information, allowing the system to adjust damping or acceleration according to the agent trajectories and the curvature of the landscape. Under suitable conditions, we prove that the underdamped system satisfies an energy dissipation law, from which we establish an upper bound on the asymptotic decay rate of the gap between the objective function and its global minimum, given by $O(1/\delta(t))$ (defined in \S 3). We further construct structure-preserving discretizations that retain both discrete energy dissipation and the convergence rate estimate, $O(1/\delta_k)$ (defined in \S3). In addition, we present several other efficient numerical algorithms for the dynamical system. Numerical experiments for all proposed algorithms validate the theory on convex test problems and demonstrate convergence rates in function values that are substantially faster than the theoretical guarantees ($O(1/\delta_k)$). On nonconvex benchmark problems, the proposed methods achieve high success rates in reaching the global minimum, and exhibit more stable energy decay than swarm-based gradient descent and Nesterov methods. Overall, this work provides a systematic framework for the construction and analysis of SBIMs from an energy-dissipative perspective.