Dynamic Estimation of Learning Rates Using a Non-Linear Autoregressive Model

TL;DR

This paper presents a novel adaptive non-linear autoregressive model that dynamically estimates learning rates and momentum, leading to stable and rapid convergence in optimization tasks, validated through extensive experiments.

Contribution

It introduces a new class of Nlar models with momentum, proposes three estimators for learning rates with proven convergence, and develops effective optimizers demonstrating robustness and adaptability.

Findings

Robust convergence despite large initial learning rates

Rapid convergence during early training epochs

Effective performance across multiple datasets and reinforcement learning environments

Abstract

We introduce a new class of adaptive non-linear autoregressive (Nlar) models incorporating the concept of momentum, which dynamically estimate both the learning rates and momentum as the number of iterations increases. In our method, the growth of the gradients is controlled using a scaling (clipping) function, leading to stable convergence. Within this framework, we propose three distinct estimators for learning rates and provide theoretical proof of their convergence. We further demonstrate how these estimators underpin the development of effective Nlar optimizers. The performance of the proposed estimators and optimizers is rigorously evaluated through extensive experiments across several datasets and a reinforcement learning environment. The results highlight two key features of the Nlar optimizers: robust convergence despite variations in underlying parameters, including large initial learning rates, and strong adaptability with rapid convergence during the initial epochs.