Learning Rate Dropout

TL;DR

Learning Rate Dropout (LRD) is a simple technique that improves neural network training by randomly dropping learning rates, encouraging exploration of the parameter space, avoiding poor local minima, and accelerating convergence.

Contribution

The paper introduces Learning Rate Dropout, a novel gradient descent method that enhances exploration and training efficiency by randomly dropping learning rates during optimization.

Findings

LRD accelerates training convergence.

LRD helps avoid saddle points and bad local minima.

LRD prevents overfitting in neural networks.

Abstract

The performance of a deep neural network is highly dependent on its training, and finding better local optimal solutions is the goal of many optimization algorithms. However, existing optimization algorithms show a preference for descent paths that converge slowly and do not seek to avoid bad local optima. In this work, we propose Learning Rate Dropout (LRD), a simple gradient descent technique for training related to coordinate descent. LRD empirically aids the optimizer to actively explore in the parameter space by randomly setting some learning rates to zero; at each iteration, only parameters whose learning rate is not 0 are updated. As the learning rate of different parameters is dropped, the optimizer will sample a new loss descent path for the current update. The uncertainty of the descent path helps the model avoid saddle points and bad local minima. Experiments show that LRD is surprisingly effective in accelerating training while preventing overfitting.