Probability-Dependent Gradient Decay in Large Margin Softmax

TL;DR

This paper introduces a probability-dependent gradient decay hyperparameter in Softmax, analyzing its impact on training dynamics, generalization, and curriculum learning, supported by theoretical insights and empirical results across multiple datasets.

Contribution

It proposes a novel gradient decay hyperparameter in Softmax, linking large margin Softmax, local Lipschitz constraints, and curriculum learning, with a dynamic warm-up strategy for training.

Findings

Gradient decay rate significantly influences generalization performance.

Small gradient decay facilitates curriculum learning by focusing on hard samples.

Dynamic adjustment of gradient decay accelerates convergence during training.

Abstract

In the past few years, Softmax has become a common component in neural network frameworks. In this paper, a gradient decay hyperparameter is introduced in Softmax to control the probability-dependent gradient decay rate during training. By following the theoretical analysis and empirical results of a variety of model architectures trained on MNIST, CIFAR-10/100 and SVHN, we find that the generalization performance depends significantly on the gradient decay rate as the confidence probability rises, i.e., the gradient decreases convexly or concavely as the sample probability increases. Moreover, optimization with the small gradient decay shows a similar curriculum learning sequence where hard samples are in the spotlight only after easy samples are convinced sufficiently, and well-separated samples gain a higher gradient to reduce intra-class distance. Based on the analysis results, we can provide evidence that the large margin Softmax will affect the local Lipschitz constraint of the loss function by regulating the probability-dependent gradient decay rate. This paper provides a new perspective and understanding of the relationship among concepts of large margin Softmax, local Lipschitz constraint and curriculum learning by analyzing the gradient decay rate. Besides, we propose a warm-up strategy to dynamically adjust Softmax loss in training, where the gradient decay rate increases from over-small to speed up the convergence rate.