SketchySGD: Reliable Stochastic Optimization via Randomized Curvature Estimates

TL;DR

SketchySGD introduces a randomized curvature estimation approach that enhances stochastic gradient methods, achieving faster convergence and better performance on ill-conditioned problems with minimal hyperparameter tuning.

Contribution

It proposes SketchySGD, a novel stochastic optimization method using randomized low-rank Hessian approximations and an automated stepsize for improved convergence.

Findings

Converges linearly to a small neighborhood of the optimum.

Outperforms traditional SGD in ill-conditioned least-squares problems.

Achieves comparable or better results than tuned stochastic methods on various datasets.

Abstract

SketchySGD improves upon existing stochastic gradient methods in machine learning by using randomized low-rank approximations to the subsampled Hessian and by introducing an automated stepsize that works well across a wide range of convex machine learning problems. We show theoretically that SketchySGD with a fixed stepsize converges linearly to a small ball around the optimum. Further, in the ill-conditioned setting we show SketchySGD converges at a faster rate than SGD for least-squares problems. We validate this improvement empirically with ridge regression experiments on real data. Numerical experiments on both ridge and logistic regression problems with dense and sparse data, show that SketchySGD equipped with its default hyperparameters can achieve comparable or better results than popular stochastic gradient methods, even when they have been tuned to yield their best performance. In particular, SketchySGD is able to solve an ill-conditioned logistic regression problem with a data matrix that takes more than $840$GB RAM to store, while its competitors, even when tuned, are unable to make any progress. SketchySGD's ability to work out-of-the box with its default hyperparameters and excel on ill-conditioned problems is an advantage over other stochastic gradient methods, most of which require careful hyperparameter tuning (especially of the learning rate) to obtain good performance and degrade in the presence of ill-conditioning.