Adaptive Step Sizes in Variance Reduction via Regularization

TL;DR

This paper introduces regularization techniques to effectively apply Barzilai-Borwein step sizes in variance reduction methods for both convex and nonconvex optimization, improving convergence and complexity bounds.

Contribution

It proposes simple regularization schemes to enable BB step sizes in non-strongly convex problems and combines them with SARAH for improved convergence in convex and nonconvex settings.

Findings

Regularized SARAH achieves better complexity bounds for convex functions.

BB step sizes are proven to converge in nonconvex problems.

Numerical tests validate the effectiveness of the proposed methods.

Abstract

The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size. With the adaptivity of BB step sizes granted, they can fail when the objective function is not strongly convex. To overcome this challenge, the key idea here is to bridge (non)convex problems and strongly convex ones via regularization. The proposed regularization schemes are \textit{simple} yet effective. Wedding the BB step size with a variance reduction method, known as SARAH, offers a free lunch compared with vanilla SARAH in convex problems. The convergence of BB step sizes in nonconvex problems is also established and its complexity is no worse than other adaptive step sizes such as AdaGrad. As a byproduct, our regularized SARAH methods for convex functions ensure that the complexity to find $\mathbb{E}[\| \nabla f(\mathbf{x}) \|^2]\leq \epsilon$ is ${\cal O}\big( (n+\frac{1}{\sqrt{\epsilon}})\ln{\frac{1}{\epsilon}}\big)$, improving $\epsilon$ dependence over existing results. Numerical tests further validate the merits of proposed approaches.