Gray Learning from Non-IID Data with Out-of-distribution Samples

TL;DR

This paper introduces Gray Learning, a novel method that leverages both ground-truth and complementary labels, with adaptive loss weighting, to improve neural network robustness on non-IID datasets containing out-of-distribution samples.

Contribution

The paper proposes Gray Learning, a new approach that uses complementary labels and adaptive loss weighting, grounded in statistical theory, to enhance learning from unreliable, non-IID data.

Findings

Gray Learning outperforms existing robust methods.

It achieves tighter generalization bounds in non-IID settings.

Experimental results show significant accuracy improvements.

Abstract

The integrity of training data, even when annotated by experts, is far from guaranteed, especially for non-IID datasets comprising both in- and out-of-distribution samples. In an ideal scenario, the majority of samples would be in-distribution, while samples that deviate semantically would be identified as out-of-distribution and excluded during the annotation process. However, experts may erroneously classify these out-of-distribution samples as in-distribution, assigning them labels that are inherently unreliable. This mixture of unreliable labels and varied data types makes the task of learning robust neural networks notably challenging. We observe that both in- and out-of-distribution samples can almost invariably be ruled out from belonging to certain classes, aside from those corresponding to unreliable ground-truth labels. This opens the possibility of utilizing reliable complementary labels that indicate the classes to which a sample does not belong. Guided by this insight, we introduce a novel approach, termed \textit{Gray Learning} (GL), which leverages both ground-truth and complementary labels. Crucially, GL adaptively adjusts the loss weights for these two label types based on prediction confidence levels. By grounding our approach in statistical learning theory, we derive bounds for the generalization error, demonstrating that GL achieves tight constraints even in non-IID settings. Extensive experimental evaluations reveal that our method significantly outperforms alternative approaches grounded in robust statistics.