PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates

TL;DR

This paper presents PROMISE, a set of preconditioned stochastic optimization algorithms with scalable curvature estimates, that outperform traditional methods in large-scale convex machine learning tasks without extensive hyperparameter tuning.

Contribution

It introduces PROMISE algorithms with theoretical guarantees, effective default hyperparameters, and a new notion of quadratic regularity for improved convergence analysis.

Findings

PROMISE algorithms outperform or match tuned optimizers on benchmark problems.

Theoretical analysis shows linear convergence under quadratic regularity.

Default hyperparameters are effective across diverse problems.

Abstract

This paper introduces PROMISE ($\textbf{Pr}$econditioned Stochastic $\textbf{O}$ptimization $\textbf{M}$ethods by $\textbf{I}$ncorporating $\textbf{S}$calable Curvature $\textbf{E}$stimates), a suite of sketching-based preconditioned stochastic gradient algorithms for solving large-scale convex optimization problems arising in machine learning. PROMISE includes preconditioned versions of SVRG, SAGA, and Katyusha; each algorithm comes with a strong theoretical analysis and effective default hyperparameter values. In contrast, traditional stochastic gradient methods require careful hyperparameter tuning to succeed, and degrade in the presence of ill-conditioning, a ubiquitous phenomenon in machine learning. Empirically, we verify the superiority of the proposed algorithms by showing that, using default hyperparameter values, they outperform or match popular tuned stochastic gradient optimizers on a test bed of $51$ ridge and logistic regression problems assembled from benchmark machine learning repositories. On the theoretical side, this paper introduces the notion of quadratic regularity in order to establish linear convergence of all proposed methods even when the preconditioner is updated infrequently. The speed of linear convergence is determined by the quadratic regularity ratio, which often provides a tighter bound on the convergence rate compared to the condition number, both in theory and in practice, and explains the fast global linear convergence of the proposed methods.