Beyond Size and Class Balance: Alpha as a New Dataset Quality Metric for Deep Learning

TL;DR

This paper introduces a new dataset quality metric called alpha, based on diversity measures from ecology, which better predicts deep learning performance than size or class balance, especially in medical imaging.

Contribution

The paper proposes alpha, a set of diversity measures, as a novel dataset quality metric that correlates more strongly with model performance than traditional size and class balance metrics.

Findings

Alpha measures explain 67% of performance variance.

Maximizing alpha improves model accuracy by up to 16%.

Size and class balance are less predictive of performance.

Abstract

In deep learning, achieving high performance on image classification tasks requires diverse training sets. However, the current best practice$\unicode{x2013}$maximizing dataset size and class balance$\unicode{x2013}$does not guarantee dataset diversity. We hypothesized that, for a given model architecture, model performance can be improved by maximizing diversity more directly. To test this hypothesis, we introduce a comprehensive framework of diversity measures from ecology that generalizes familiar quantities like Shannon entropy by accounting for similarities among images. (Size and class balance emerge as special cases.) Analyzing thousands of subsets from seven medical datasets showed that the best correlates of performance were not size or class balance but $A$$\unicode{x2013}$"big alpha"$\unicode{x2013}$a set of generalized entropy measures interpreted as the effective number of image-class pairs in the dataset, after accounting for image similarities. One of these, $A_0$, explained 67% of the variance in balanced accuracy, vs. 54% for class balance and just 39% for size. The best pair of measures was size-plus-$A_1$ (79%), which outperformed size-plus-class-balance (74%). Subsets with the largest $A_0$ performed up to 16% better than those with the largest size (median improvement, 8%). We propose maximizing $A$ as a way to improve deep learning performance in medical imaging.