SPI-Optimizer: an integral-Separated PI Controller for Stochastic Optimization

TL;DR

The paper introduces SPI-Optimizer, a novel integral-Separated PI controller-based optimizer that reduces oscillation and accelerates convergence in stochastic optimization without extra hyperparameters, outperforming existing methods.

Contribution

It proposes a hyperparameter-free, integral-Separated PI controller for optimization, addressing oscillation issues and improving convergence speed and accuracy.

Findings

Eliminates oscillation in stochastic optimization.

Reduces training epochs by up to 40%.

Achieves up to 27.5% error reduction on benchmarks.

Abstract

To overcome the oscillation problem in the classical momentum-based optimizer, recent work associates it with the proportional-integral (PI) controller, and artificially adds D term producing a PID controller. It suppresses oscillation with the sacrifice of introducing extra hyper-parameter. In this paper, we start by analyzing: why momentum-based method oscillates about the optimal point? and answering that: the fluctuation problem relates to the lag effect of integral (I) term. Inspired by the conditional integration idea in classical control society, we propose SPI-Optimizer, an integral-Separated PI controller based optimizer WITHOUT introducing extra hyperparameter. It separates momentum term adaptively when the inconsistency of current and historical gradient direction occurs. Extensive experiments demonstrate that SPIOptimizer generalizes well on popular network architectures to eliminate the oscillation, and owns competitive performance with faster convergence speed (up to 40% epochs reduction ratio ) and more accurate classification result on MNIST, CIFAR10, and CIFAR100 (up to 27.5% error reduction ratio) than the state-of-the-art methods.