Pre-train to Gain: Robust Learning Without Clean Labels

TL;DR

Pre-training with self-supervised learning before supervised training enhances robustness to noisy labels in deep networks, eliminating the need for clean data subsets and improving accuracy especially under high noise conditions.

Contribution

The paper introduces a noise-robust training method using self-supervised pre-training, which outperforms traditional approaches that require clean data subsets.

Findings

Self-supervised pre-training improves classification accuracy across all noise levels.

The approach enhances label-error detection metrics such as F1 and Balanced Accuracy.

Performance surpasses ImageNet pre-trained models under high noise conditions.

Abstract

Training deep networks with noisy labels leads to poor generalization and degraded accuracy due to overfitting to label noise. Existing approaches for learning with noisy labels often rely on the availability of a clean subset of data. By pre-training a feature extractor backbone without labels using self-supervised learning (SSL), followed by standard supervised training on the noisy dataset, we can train a more noise robust model without requiring a subset with clean labels. We evaluate the use of SimCLR and Barlow~Twins as SSL methods on CIFAR-10 and CIFAR-100 under synthetic and real world noise. Across all noise rates, self-supervised pre-training consistently improves classification accuracy and enhances downstream label-error detection (F1 and Balanced Accuracy). The performance gap widens as the noise rate increases, demonstrating improved robustness. Notably, our approach achieves comparable results to ImageNet pre-trained models at low noise levels, while substantially outperforming them under high noise conditions.