Geometric Median (GM) Matching for Robust Data Pruning

TL;DR

This paper introduces Geometric Median Matching, a robust data pruning method that effectively handles noisy datasets by approximating the geometric median, outperforming previous techniques especially under high corruption.

Contribution

We propose a novel greedy algorithm for robust data pruning that achieves optimal breakdown point and improved scaling over uniform sampling.

Findings

Outperforms prior methods in noisy data scenarios

Achieves optimal breakdown point of 1/2 under arbitrary corruption

Shows significant improvements at high corruption and pruning rates

Abstract

Large-scale data collections in the wild, are invariably noisy. Thus developing data pruning strategies that remain robust even in the presence of corruption is critical in practice. In this work, we propose Geometric Median ($\gm$) Matching -- a herding style greedy algorithm that yields a $k$-subset such that the mean of the subset approximates the geometric median of the (potentially) noisy dataset. Theoretically, we show that $\gm$ Matching enjoys an improved $\gO(1/k)$ scaling over $\gO(1/\sqrt{k})$ scaling of uniform sampling; while achieving {\bf optimal breakdown point} of {\bf 1/2} even under {\bf arbitrary} corruption. Extensive experiments across several popular deep learning benchmarks indicate that $\gm$ Matching consistently improves over prior state-of-the-art; the gains become more profound at high rates of corruption and aggressive pruning rates; making $\gm$ Matching a strong baseline for future research in robust data pruning.