Removing Undesirable Feature Contributions Using Out-of-Distribution Data

TL;DR

This paper introduces a novel data augmentation approach using out-of-distribution data to enhance neural network generalization, overcoming limitations of in-distribution augmentation methods and improving adversarial training performance.

Contribution

The paper proposes a new augmentation method leveraging OOD data, with theoretical analysis and empirical validation on multiple datasets, outperforming existing UID-based methods.

Findings

Improves generalization in adversarial and standard learning scenarios.

Outperforms existing UID-based augmentation methods.

Enhances state-of-the-art adversarial training.

Abstract

Several data augmentation methods deploy unlabeled-in-distribution (UID) data to bridge the gap between the training and inference of neural networks. However, these methods have clear limitations in terms of availability of UID data and dependence of algorithms on pseudo-labels. Herein, we propose a data augmentation method to improve generalization in both adversarial and standard learning by using out-of-distribution (OOD) data that are devoid of the abovementioned issues. We show how to improve generalization theoretically using OOD data in each learning scenario and complement our theoretical analysis with experiments on CIFAR-10, CIFAR-100, and a subset of ImageNet. The results indicate that undesirable features are shared even among image data that seem to have little correlation from a human point of view. We also present the advantages of the proposed method through comparison with other data augmentation methods, which can be used in the absence of UID data. Furthermore, we demonstrate that the proposed method can further improve the existing state-of-the-art adversarial training.