Training With Data Dependent Dynamic Learning Rates

TL;DR

This paper introduces a novel optimization framework that learns dynamic, instance-specific learning rates for training deep neural networks, improving performance especially on noisy data and enabling model personalization.

Contribution

It proposes a new optimizer that adapts learning rates per data instance, relaxing the assumption of uniform loss characteristics across data, and demonstrates its effectiveness across tasks.

Findings

Consistent performance gains across CNN architectures.

Improved handling of noisy, corrupt data instances.

Enables model personalization towards specific data distributions.

Abstract

Recently many first and second order variants of SGD have been proposed to facilitate training of Deep Neural Networks (DNNs). A common limitation of these works stem from the fact that they use the same learning rate across all instances present in the dataset. This setting is widely adopted under the assumption that loss functions for each instance are similar in nature, and hence, a common learning rate can be used. In this work, we relax this assumption and propose an optimization framework which accounts for difference in loss function characteristics across instances. More specifically, our optimizer learns a dynamic learning rate for each instance present in the dataset. Learning a dynamic learning rate for each instance allows our optimization framework to focus on different modes of training data during optimization. When applied to an image classification task, across different CNN architectures, learning dynamic learning rates leads to consistent gains over standard optimizers. When applied to a dataset containing corrupt instances, our framework reduces the learning rates on noisy instances, and improves over the state-of-the-art. Finally, we show that our optimization framework can be used for personalization of a machine learning model towards a known targeted data distribution.