AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio

TL;DR

AdaSmooth is a new adaptive learning rate method for gradient descent that reduces the need for hyper-parameter tuning and performs well across various neural network architectures and tasks.

Contribution

It introduces a hyper-parameter insensitive, per-dimension learning rate method called AdaSmooth for stochastic optimization.

Findings

AdaSmooth outperforms other methods on CNNs and MLPs.

It requires no manual hyper-parameter tuning.

Empirical results show strong practical performance.

Abstract

It is well known that we need to choose the hyper-parameters in Momentum, AdaGrad, AdaDelta, and other alternative stochastic optimizers. While in many cases, the hyper-parameters are tuned tediously based on experience becoming more of an art than science. We present a novel per-dimension learning rate method for gradient descent called AdaSmooth. The method is insensitive to hyper-parameters thus it requires no manual tuning of the hyper-parameters like Momentum, AdaGrad, and AdaDelta methods. We show promising results compared to other methods on different convolutional neural networks, multi-layer perceptron, and alternative machine learning tasks. Empirical results demonstrate that AdaSmooth works well in practice and compares favorably to other stochastic optimization methods in neural networks.