Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent

TL;DR

This paper introduces Meta-Regularization, a novel method for adaptively choosing learning rates in gradient descent by incorporating a regularization term, leading to algorithms with competitive convergence and improved performance.

Contribution

It presents a new regularization-based framework for adaptive learning rate selection, including theoretical analysis and practical algorithms using ivergence regularizers.

Findings

Algorithms achieve comparable convergence to existing methods.

Certain regularizers can enhance convergence under strong convexity.

Numerical experiments validate effectiveness in benchmark and online settings.

Abstract

We propose \textit{Meta-Regularization}, a novel approach for the adaptive choice of the learning rate in first-order gradient descent methods. Our approach modifies the objective function by adding a regularization term on the learning rate, and casts the joint updating process of parameters and learning rates into a maxmin problem. Given any regularization term, our approach facilitates the generation of practical algorithms. When \textit{Meta-Regularization} takes the $\varphi$-divergence as a regularizer, the resulting algorithms exhibit comparable theoretical convergence performance with other first-order gradient-based algorithms. Furthermore, we theoretically prove that some well-designed regularizers can improve the convergence performance under the strong-convexity condition of the objective function. Numerical experiments on benchmark problems demonstrate the effectiveness of algorithms derived from some common $\varphi$-divergence in full batch as well as online learning settings.