Why Can Accurate Models Be Learned from Inaccurate Annotations?

TL;DR

This paper investigates why models trained on noisy, inaccurate labels can still make accurate predictions, revealing that principal subspace alignment explains robustness and proposing a method to improve it.

Contribution

The paper provides a theoretical analysis of label noise effects on model weights and introduces LIP, a plug-in to preserve principal subspace information under label inaccuracies.

Findings

Principal subspace remains largely aligned despite label noise

Angles of principal subspaces show minimal deviation with moderate noise

LIP improves model robustness across various noise conditions

Abstract

Learning from inaccurate annotations has gained significant attention due to the high cost of precise labeling. However, despite the presence of erroneous labels, models trained on noisy data often retain the ability to make accurate predictions. This intriguing phenomenon raises a fundamental yet largely unexplored question: why models can still extract correct label information from inaccurate annotations remains unexplored. In this paper, we conduct a comprehensive investigation into this issue. By analyzing weight matrices from both empirical and theoretical perspectives, we find that label inaccuracy primarily accumulates noise in lower singular components and subtly perturbs the principal subspace. Within a certain range, the principal subspaces of weights trained on inaccurate labels remain largely aligned with those learned from clean labels, preserving essential task-relevant information. We formally prove that the angles of principal subspaces exhibit minimal deviation under moderate label inaccuracy, explaining why models can still generalize effectively. Building on these insights, we propose LIP, a lightweight plug-in designed to help classifiers retain principal subspace information while mitigating noise induced by label inaccuracy. Extensive experiments on tasks with various inaccuracy conditions demonstrate that LIP consistently enhances the performance of existing algorithms. We hope our findings can offer valuable theoretical and practical insights to understand of model robustness under inaccurate supervision.