High-dimensional Learning with Noisy Labels

TL;DR

This paper analyzes high-dimensional binary classification with noisy labels, revealing that low-dimensional intuitions fail in high dimensions and proposing an optimized method that outperforms baselines.

Contribution

It provides a theoretical framework for understanding noisy label effects in high dimensions and introduces an optimized classifier that improves performance.

Findings

High-dimensional label noise impacts classifiers differently than in low dimensions.

The proposed method outperforms baseline classifiers on real datasets.

Theoretical analysis confirms the effectiveness of the optimized approach.

Abstract

This paper provides theoretical insights into high-dimensional binary classification with class-conditional noisy labels. Specifically, we study the behavior of a linear classifier with a label noisiness aware loss function, when both the dimension of data $p$ and the sample size $n$ are large and comparable. Relying on random matrix theory by supposing a Gaussian mixture data model, the performance of the linear classifier when $p,n\to \infty$ is shown to converge towards a limit, involving scalar statistics of the data. Importantly, our findings show that the low-dimensional intuitions to handle label noise do not hold in high-dimension, in the sense that the optimal classifier in low-dimension dramatically fails in high-dimension. Based on our derivations, we design an optimized method that is shown to be provably more efficient in handling noisy labels in high dimensions. Our theoretical conclusions are further confirmed by experiments on real datasets, where we show that our optimized approach outperforms the considered baselines.