Particle Filtering Methods for Stochastic Optimization with Application to Large-Scale Empirical Risk Minimization

TL;DR

This paper introduces particle filtering-based stochastic optimization methods that outperform traditional approaches like SGD and Kalman filter-based methods in large-scale empirical risk minimization tasks, especially for nonlinear problems.

Contribution

The paper develops two novel particle filtering-based stochastic optimizers that overcome limitations of Kalman filter-based methods in nonlinear, non-Gaussian settings.

Findings

PFSOs outperform SGD, IPM, and KF-based methods in stability and speed.

PFSOs handle diverse loss functions effectively.

Experimental results confirm superior convergence and flexibility.

Abstract

This paper is concerned with sequential filtering based stochastic optimization (FSO) approaches that leverage a probabilistic perspective to implement the incremental proximity method (IPM). The present FSO methods are derived based on the Kalman filter (KF) and the extended KF (EKF). In contrast with typical methods such as stochastic gradient descent (SGD) and IPMs, they do not need to pre-schedule the learning rate for convergence. Nevertheless, they have limitations that inherit from the KF mechanism. As the particle filtering (PF) method outperforms KF and its variants remarkably for nonlinear non-Gaussian sequential filtering problems, it is natural to ask if FSO methods can benefit from PF to get around of their limitations. We provide an affirmative answer to this question by developing two PF based stochastic optimizers (PFSOs). For performance evaluation, we apply them to address nonlinear least-square fitting with simulated data, and empirical risk minimization for binary classification of real data sets. Experimental results demonstrate that PFSOs outperform remarkably a benchmark SGD algorithm, the vanilla IPM, and KF-type FSO methods in terms of numerical stability, convergence speed, and flexibility in handling diverse types of loss functions.