QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning

TL;DR

QLABGrad is a novel, hyperparameter-free learning rate adaptation scheme for deep learning that guarantees convergence and outperforms existing methods across multiple architectures and datasets.

Contribution

It introduces QLABGrad, a hyperparameter-free scheme that automatically adapts learning rates with proven convergence guarantees for deep learning.

Findings

Outperforms competing schemes on multiple architectures.

Proven convergence under Lipschitz condition.

Effective across MNIST, CIFAR10, and ImageNet datasets.

Abstract

The learning rate is a critical hyperparameter for deep learning tasks since it determines the extent to which the model parameters are updated during the learning course. However, the choice of learning rates typically depends on empirical judgment, which may not result in satisfactory outcomes without intensive try-and-error experiments. In this study, we propose a novel learning rate adaptation scheme called QLABGrad. Without any user-specified hyperparameter, QLABGrad automatically determines the learning rate by optimizing the Quadratic Loss Approximation-Based (QLAB) function for a given gradient descent direction, where only one extra forward propagation is required. We theoretically prove the convergence of QLABGrad with a smooth Lipschitz condition on the loss function. Experiment results on multiple architectures, including MLP, CNN, and ResNet, on MNIST, CIFAR10, and ImageNet datasets, demonstrate that QLABGrad outperforms various competing schemes for deep learning.