Adaptive Gradient Method with Resilience and Momentum

TL;DR

This paper introduces AdaRem, an adaptive gradient method that improves training speed and generalization in deep neural networks by reducing oscillations through a novel parameter-wise learning rate adjustment based on gradient direction consistency.

Contribution

AdaRem is a new adaptive gradient method that encourages long-term consistent updates, leading to faster convergence and better generalization compared to existing methods.

Findings

AdaRem outperforms previous adaptive methods in training speed.

AdaRem achieves lower test error on ImageNet.

Theoretical convergence of AdaRem is established.

Abstract

Several variants of stochastic gradient descent (SGD) have been proposed to improve the learning effectiveness and efficiency when training deep neural networks, among which some recent influential attempts would like to adaptively control the parameter-wise learning rate (e.g., Adam and RMSProp). Although they show a large improvement in convergence speed, most adaptive learning rate methods suffer from compromised generalization compared with SGD. In this paper, we proposed an Adaptive Gradient Method with Resilience and Momentum (AdaRem), motivated by the observation that the oscillations of network parameters slow the training, and give a theoretical proof of convergence. For each parameter, AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient, and thus encourages long-term consistent parameter updating with much fewer oscillations. Comprehensive experiments have been conducted to verify the effectiveness of AdaRem when training various models on a large-scale image recognition dataset, e.g., ImageNet, which also demonstrate that our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error, respectively.