An Embedding is Worth a Thousand Noisy Labels

TL;DR

This paper introduces WANN, a robust and efficient weighted nearest neighbor method that leverages foundation models and reliability scores to improve deep learning performance under noisy labels, reducing data requirements and enhancing generalization.

Contribution

WANN is a novel weighted adaptive nearest neighbor approach that effectively handles label noise using self-supervised features and reliability scores, outperforming existing methods in accuracy and efficiency.

Findings

WANN outperforms reference methods across diverse datasets and noise conditions.

WANN improves generalization on imbalanced data compared to other nearest neighbor methods.

The weighting scheme enhances supervised dimensionality reduction with noisy labels.

Abstract

The performance of deep neural networks scales with dataset size and label quality, rendering the efficient mitigation of low-quality data annotations crucial for building robust and cost-effective systems. Existing strategies to address label noise exhibit severe limitations due to computational complexity and application dependency. In this work, we propose WANN, a Weighted Adaptive Nearest Neighbor approach that builds on self-supervised feature representations obtained from foundation models. To guide the weighted voting scheme, we introduce a reliability score $\eta$, which measures the likelihood of a data label being correct. WANN outperforms reference methods, including a linear layer trained with robust loss functions, on diverse datasets of varying size and under various noise types and severities. WANN also exhibits superior generalization on imbalanced data compared to both Adaptive-NNs (ANN) and fixed k-NNs. Furthermore, the proposed weighting scheme enhances supervised dimensionality reduction under noisy labels. This yields a significant boost in classification performance with 10x and 100x smaller image embeddings, minimizing latency and storage requirements. Our approach, emphasizing efficiency and explainability, emerges as a simple, robust solution to overcome inherent limitations of deep neural network training. The code is available at https://github.com/francescodisalvo05/wann-noisy-labels .