Adaptive Regularization of Labels

TL;DR

This paper introduces an adaptive label regularization method that allows neural networks to learn from label errors and update labels online, improving performance across image and text classification tasks.

Contribution

The proposed method is a novel adaptive label regularization technique that updates label representations during training without requiring a teacher network.

Findings

Significant accuracy improvements on CIFAR-10, CIFAR-100, ImageNet, AGNews, Yahoo, and Yelp-Full datasets.

Requires minimal additional parameters compared to knowledge distillation.

Effective in both image recognition and text classification tasks.

Abstract

Recently, a variety of regularization techniques have been widely applied in deep neural networks, such as dropout, batch normalization, data augmentation, and so on. These methods mainly focus on the regularization of weight parameters to prevent overfitting effectively. In addition, label regularization techniques such as label smoothing and label disturbance have also been proposed with the motivation of adding a stochastic perturbation to labels. In this paper, we propose a novel adaptive label regularization method, which enables the neural network to learn from the erroneous experience and update the optimal label representation online. On the other hand, compared with knowledge distillation, which learns the correlation of categories using teacher network, our proposed method requires only a minuscule increase in parameters without cumbersome teacher network. Furthermore, we evaluate our method on CIFAR-10/CIFAR-100/ImageNet datasets for image recognition tasks and AGNews/Yahoo/Yelp-Full datasets for text classification tasks. The empirical results show significant improvement under all experimental settings.