Stochastic Gradient Descent with Preconditioned Polyak Step-size

TL;DR

This paper introduces a preconditioned extension of the SPS method for SGD, aiming to enhance optimization performance on ill-conditioned datasets without the need for extensive learning rate tuning.

Contribution

It proposes a novel preconditioned SPS algorithm that integrates techniques like Hutchinson's, Adam, and AdaGrad to improve robustness and efficiency.

Findings

Improved convergence on ill-conditioned datasets

Reduced need for learning rate tuning

Enhanced performance with preconditioning techniques

Abstract

Stochastic Gradient Descent (SGD) is one of the many iterative optimization methods that are widely used in solving machine learning problems. These methods display valuable properties and attract researchers and industrial machine learning engineers with their simplicity. However, one of the weaknesses of this type of methods is the necessity to tune learning rate (step-size) for every loss function and dataset combination to solve an optimization problem and get an efficient performance in a given time budget. Stochastic Gradient Descent with Polyak Step-size (SPS) is a method that offers an update rule that alleviates the need of fine-tuning the learning rate of an optimizer. In this paper, we propose an extension of SPS that employs preconditioning techniques, such as Hutchinson's method, Adam, and AdaGrad, to improve its performance on badly scaled and/or ill-conditioned datasets.