Stable gradient-adjusted root mean square propagation on least squares problem

TL;DR

This paper introduces a stable gradient-adjusted RMSProp algorithm for least squares problems, demonstrating its convergence properties and superior initial convergence compared to SGD and RMSProp, with an adaptive switching strategy.

Contribution

The paper proposes SGA-RMSProp, a novel stable variant of RMSProp with proven convergence on least squares problems and an adaptive method to switch between algorithms.

Findings

SGA-RMSProp converges R-linearly on consistent problems.

It converges to a neighborhood of the minimizer in inconsistent cases.

Numerical results show faster initial convergence of SGA-RMSProp.

Abstract

Root mean square propagation (abbreviated as RMSProp) is a first-order stochastic algorithm used in machine learning widely. In this paper, a stable gradient-adjusted RMSProp (abbreviated as SGA-RMSProp) with mini-batch stochastic gradient is proposed, and its properties are studied on the linear least squares problem. R-linear convergence of the algorithm is established on the consistent linear least squares problem. The algorithm is also proved to converge R-linearly to a neighborhood of the minimizer for the inconsistent case, with the region of the neighborhood being controlled by the batch size. Furthermore, numerical experiments are conducted to compare the performances of SGA-RMSProp, stochastic gradient descent (abbreviated as SGD), and the original RMSProp with different batch sizes. The faster initial convergence rate of SGA-RMSProp is observed through numerical experiments and an adaptive strategy for switching from SGA-RMSProp to SGD is proposed, which combines the benefits of these two algorithms.