LALR: Theoretical and Experimental validation of Lipschitz Adaptive Learning Rate in Regression and Neural Networks

TL;DR

This paper introduces a Lipschitz-based adaptive learning rate framework for regression and neural networks, demonstrating significantly faster convergence through theoretical analysis and experimental validation.

Contribution

It presents a novel adaptive learning rate method grounded in Lipschitz continuity theory, validated for regression tasks with neural networks.

Findings

Up to 20x faster convergence with the adaptive policy

Effective for Mean Absolute Error and Quantile loss functions

Theoretical and experimental validation of the approach

Abstract

We propose a theoretical framework for an adaptive learning rate policy for the Mean Absolute Error loss function and Quantile loss function and evaluate its effectiveness for regression tasks. The framework is based on the theory of Lipschitz continuity, specifically utilizing the relationship between learning rate and Lipschitz constant of the loss function. Based on experimentation, we have found that the adaptive learning rate policy enables up to 20x faster convergence compared to a constant learning rate policy.