Leveraging Data Symmetries to Select an Optimal Subset of Training Data under Label Noise

TL;DR

This paper explores how leveraging data symmetries and invariances can improve subset selection for training machine learning models in noisy, high-dimensional data environments, enhancing robustness and performance.

Contribution

It formally links k-NN accuracy to classifier performance on noisy data and demonstrates that exploiting data invariances improves subset selection in high-dimensional settings.

Findings

Exploiting data invariance enhances k-NN performance in noisy environments.

Invariance knowledge helps identify near-optimal training subsets.

Learned invariant representations facilitate subset selection with partial invariance knowledge.

Abstract

The performance of machine learning models often relies on large labeled datasets; however, data collected from diverse sources can contain label noise. Recent work has shown that, in noisy settings, there may exist a subset of the training data on which models can achieve performance comparable to training on a noise-free dataset. A widely used method for identifying such subsets is cutstats, which employs k-nearest neighbors (k-NN) to detect low-noise samples. However, its performance on high-dimensional data remains largely unexplored. In this work, we formally establish that the performance of a classifier trained on a subset of a noisy dataset selected via cutstats is influenced by the accuracy of k-NN. We further demonstrate that, in noisy environments, exploiting data invariance and knowledge of underlying symmetries can significantly enhance the performance of k-NN, bringing it closer to the Bayes optimal classifier even in high-dimensional regimes. Finally, we show that for real-world scenarios, where information about the underlying invariance is only partially known, learnt invariant representations can still facilitate the identification of near-optimal subsets.