Dataset Diversity Metrics and Impact on Classification Models

TL;DR

This paper investigates how various dataset diversity metrics relate to model performance and real-world clinical variability, revealing limited correlation with some metrics and highlighting scanner diversity as a key factor.

Contribution

It systematically evaluates multiple diversity metrics across image and medical datasets, analyzing their correlation with model performance and clinical expert insights.

Findings

Limited correlation between AUC and reference-free diversity metrics

Higher correlation between FID, semantic diversity, and model performance

Adding scanners can cause shortcut learning in models

Abstract

The diversity of training datasets is usually perceived as an important aspect to obtain a robust model. However, the definition of diversity is often not defined or differs across papers, and while some metrics exist, the quantification of this diversity is often overlooked when developing new algorithms. In this work, we study the behaviour of multiple dataset diversity metrics for image, text and metadata using MorphoMNIST, a toy dataset with controlled perturbations, and PadChest, a publicly available chest X-ray dataset. We evaluate whether these metrics correlate with each other but also with the intuition of a clinical expert. We also assess whether they correlate with downstream-task performance and how they impact the training dynamic of the models. We find limited correlations between the AUC and image or metadata reference-free diversity metrics, but higher correlations with the FID and the semantic diversity metrics. Finally, the clinical expert indicates that scanners are the main source of diversity in practice. However, we find that the addition of another scanner to the training set leads to shortcut learning. The code used in this study is available at https://github.com/TheoSourget/dataset_diversity_evaluation