Forget the Learning Rate, Decay Loss

TL;DR

This paper proposes a novel loss decay strategy that replaces traditional learning rate schedules, demonstrating improved performance across various deep learning tasks like image classification, segmentation, and GANs.

Contribution

It introduces a fixed learning rate approach combined with loss decay to enhance model training without adjusting the learning rate.

Findings

Loss decay strategy significantly improves model accuracy.

Method is effective across multiple tasks including classification, segmentation, and GANs.

Simplifies training by removing the need for learning rate scheduling.

Abstract

In the usual deep neural network optimization process, the learning rate is the most important hyper parameter, which greatly affects the final convergence effect. The purpose of learning rate is to control the stepsize and gradually reduce the impact of noise on the network. In this paper, we will use a fixed learning rate with method of decaying loss to control the magnitude of the update. We used Image classification, Semantic segmentation, and GANs to verify this method. Experiments show that the loss decay strategy can greatly improve the performance of the model