Selective Output Smoothing Regularization: Regularize Neural Networks by Softening Output Distributions

TL;DR

This paper introduces Selective Output Smoothing Regularization, a simple yet effective method to improve CNN training by encouraging more balanced output distributions on correctly classified samples, leading to better accuracy.

Contribution

The paper proposes a novel regularization technique that can be easily integrated into CNN training to enhance performance across multiple image classification benchmarks.

Findings

Achieves 77.30% accuracy on ImageNet with ResNet-50, outperforming baseline by 1.1%.

Consistently improves results on CIFAR-100, Tiny ImageNet, and CUB-200-2011.

Enhances detection performance when used with pretrained models on Pascal detection.

Abstract

In this paper, we propose Selective Output Smoothing Regularization, a novel regularization method for training the Convolutional Neural Networks (CNNs). Inspired by the diverse effects on training from different samples, Selective Output Smoothing Regularization improves the performance by encouraging the model to produce equal logits on incorrect classes when dealing with samples that the model classifies correctly and over-confidently. This plug-and-play regularization method can be conveniently incorporated into almost any CNN-based project without extra hassle. Extensive experiments have shown that Selective Output Smoothing Regularization consistently achieves significant improvement in image classification benchmarks, such as CIFAR-100, Tiny ImageNet, ImageNet, and CUB-200-2011. Particularly, our method obtains 77.30% accuracy on ImageNet with ResNet-50, which gains 1.1% than baseline (76.2%). We also empirically demonstrate the ability of our method to make further improvements when combining with other widely used regularization techniques. On Pascal detection, using the SOSR-trained ImageNet classifier as the pretrained model leads to better detection performances.