A Particle-based Sparse Gaussian Process Optimizer

TL;DR

This paper introduces a particle-swarm-based optimization framework that employs Gaussian Process Regression to enhance exploration and escape local minima in non-convex and high-dimensional optimization problems, outperforming traditional methods.

Contribution

It presents a novel particle-swarm-based optimizer using Gaussian Process Regression to dynamically learn descent directions, improving exploration and avoiding local minima.

Findings

Outperforms state-of-the-art optimizers in escaping local minima.

Effective in high-dimensional tasks like image classification.

Demonstrates improved exploration in non-convex optimization.

Abstract

Task learning in neural networks typically requires finding a globally optimal minimizer to a loss function objective. Conventional designs of swarm based optimization methods apply a fixed update rule, with possibly an adaptive step-size for gradient descent based optimization. While these methods gain huge success in solving different optimization problems, there are some cases where these schemes are either inefficient or suffering from local-minimum. We present a new particle-swarm-based framework utilizing Gaussian Process Regression to learn the underlying dynamical process of descent. The biggest advantage of this approach is greater exploration around the current state before deciding a descent direction. Empirical results show our approach can escape from the local minima compare with the widely-used state-of-the-art optimizers when solving non-convex optimization problems. We also test our approach under high-dimensional parameter space case, namely, image classification task.