Towards an Analytical Definition of Sufficient Data

TL;DR

This paper identifies which training samples are most informative for classifying data by analyzing their position relative to class centroids, showing that samples farther from the centroid are more valuable, and that small data reductions near centroids do not harm performance.

Contribution

It introduces a method to identify informative samples based on their position in reduced space, revealing that samples near centroids are less critical for training.

Findings

Samples closer to class centroids are less informative.

Excluding up to 2% of data near centroids does not significantly affect performance.

The method applies across datasets of varying complexity.

Abstract

We show that, for each of five datasets of increasing complexity, certain training samples are more informative of class membership than others. These samples can be identified a priori to training by analyzing their position in reduced dimensional space relative to the classes' centroids. Specifically, we demonstrate that samples nearer the classes' centroids are less informative than those that are furthest from it. For all five datasets, we show that there is no statistically significant difference between training on the entire training set and when excluding up to 2% of the data nearest to each class's centroid.