No learning rates needed: Introducing SALSA -- Stable Armijo Line Search Adaptation

TL;DR

This paper introduces SALSA, an improved Armijo line search method that eliminates the need for learning rate tuning in stochastic optimization, demonstrating superior performance across diverse models and datasets.

Contribution

The paper proposes enhancements to the Armijo line search, including faster computation and momentum integration, making it more effective for stochastic mini-batch training.

Findings

Outperforms previous Armijo implementation and tuned learning rate schedules

Effective across diverse architectures like Transformers, CNNs, MLPs

Works well on NLP and image data domains

Abstract

In recent studies, line search methods have been demonstrated to significantly enhance the performance of conventional stochastic gradient descent techniques across various datasets and architectures, while making an otherwise critical choice of learning rate schedule superfluous. In this paper, we identify problems of current state-of-the-art of line search methods, propose enhancements, and rigorously assess their effectiveness. Furthermore, we evaluate these methods on orders of magnitude larger datasets and more complex data domains than previously done. More specifically, we enhance the Armijo line search method by speeding up its computation and incorporating a momentum term into the Armijo criterion, making it better suited for stochastic mini-batching. Our optimization approach outperforms both the previous Armijo implementation and a tuned learning rate schedule for the Adam and SGD optimizers. Our evaluation covers a diverse range of architectures, such as Transformers, CNNs, and MLPs, as well as data domains, including NLP and image data. Our work is publicly available as a Python package, which provides a simple Pytorch optimizer.