Hamiltonian Monte Carlo Particle Swarm Optimizer

TL;DR

The paper presents HMC-PSO, a novel optimization algorithm combining Hamiltonian dynamics with particle swarm principles, enhancing exploration and exploitation especially for non-convex functions and neural network training.

Contribution

It introduces HMC-PSO, integrating Hamiltonian Monte Carlo with PSO, and extends it to approximate gradients for deep neural network optimization.

Findings

HMC-PSO effectively explores non-convex functions.

It outperforms some state-of-the-art optimizers on benchmark tasks.

The method provides a new approach to gradient approximation in DNNs.

Abstract

We introduce the Hamiltonian Monte Carlo Particle Swarm Optimizer (HMC-PSO), an optimization algorithm that reaps the benefits of both Exponentially Averaged Momentum PSO and HMC sampling. The coupling of the position and velocity of each particle with Hamiltonian dynamics in the simulation allows for extensive freedom for exploration and exploitation of the search space. It also provides an excellent technique to explore highly non-convex functions while ensuring efficient sampling. We extend the method to approximate error gradients in closed form for Deep Neural Network (DNN) settings. We discuss possible methods of coupling and compare its performance to that of state-of-the-art optimizers on the Golomb's Ruler problem and Classification tasks.