Adaptive Learning Rate and Momentum for Training Deep Neural Networks

TL;DR

This paper introduces a novel adaptive training method for deep neural networks based on a nonlinear Conjugate Gradient framework, which automatically adjusts hyperparameters for faster convergence and better generalization.

Contribution

The paper proposes the CGQ method that adaptively updates learning rate and momentum, eliminating the need for manual hyperparameter tuning during training.

Findings

Faster convergence compared to existing local solvers

Improved test set accuracy indicating better generalization

Theoretically proven convergence in strong convex settings

Abstract

Recent progress on deep learning relies heavily on the quality and efficiency of training algorithms. In this paper, we develop a fast training method motivated by the nonlinear Conjugate Gradient (CG) framework. We propose the Conjugate Gradient with Quadratic line-search (CGQ) method. On the one hand, a quadratic line-search determines the step size according to current loss landscape. On the other hand, the momentum factor is dynamically updated in computing the conjugate gradient parameter (like Polak-Ribiere). Theoretical results to ensure the convergence of our method in strong convex settings is developed. And experiments in image classification datasets show that our method yields faster convergence than other local solvers and has better generalization capability (test set accuracy). One major advantage of the paper method is that tedious hand tuning of hyperparameters like the learning rate and momentum is avoided.