Generalization by Recognizing Confusion

TL;DR

This paper introduces a method combining self-adaptive training with mixup to improve neural network accuracy and robustness against label noise, leading to better generalization and state-of-the-art results.

Contribution

The paper proposes a novel combination of self-adaptive training and mixup, demonstrating improved accuracy, robustness, and generalization in neural networks for image recognition.

Findings

Achieves state-of-the-art accuracy on noisy datasets

Shows low Rademacher complexity indicating provable generalization

Links rare class difficulty to robustness under label noise

Abstract

A recently-proposed technique called self-adaptive training augments modern neural networks by allowing them to adjust training labels on the fly, to avoid overfitting to samples that may be mislabeled or otherwise non-representative. By combining the self-adaptive objective with mixup, we further improve the accuracy of self-adaptive models for image recognition; the resulting classifier obtains state-of-the-art accuracies on datasets corrupted with label noise. Robustness to label noise implies a lower generalization gap; thus, our approach also leads to improved generalizability. We find evidence that the Rademacher complexity of these algorithms is low, suggesting a new path towards provable generalization for this type of deep learning model. Last, we highlight a novel connection between difficulties accounting for rare classes and robustness under noise, as rare classes are in a sense indistinguishable from label noise. Our code can be found at https://github.com/Tuxianeer/generalizationconfusion.