Exploring the Impact of Dataset Statistical Effect Size on Model Performance and Data Sample Size Sufficiency

TL;DR

This study investigates whether basic statistical effect size measures can predict model performance and data sufficiency, finding that they are not reliable heuristics for these purposes.

Contribution

The paper empirically tests the correlation between effect size and model success, revealing limitations of effect size as a predictive tool for data adequacy.

Findings

Effect size does not reliably predict model performance.

Effect size does not correlate with convergence rate or sample size needs.

Additional methods are needed for prospective data adequacy assessment.

Abstract

Having a sufficient quantity of quality data is a critical enabler of training effective machine learning models. Being able to effectively determine the adequacy of a dataset prior to training and evaluating a model's performance would be an essential tool for anyone engaged in experimental design or data collection. However, despite the need for it, the ability to prospectively assess data sufficiency remains an elusive capability. We report here on two experiments undertaken in an attempt to better ascertain whether or not basic descriptive statistical measures can be indicative of how effective a dataset will be at training a resulting model. Leveraging the effect size of our features, this work first explores whether or not a correlation exists between effect size, and resulting model performance (theorizing that the magnitude of the distinction between classes could correlate to a classifier's resulting success). We then explore whether or not the magnitude of the effect size will impact the rate of convergence of the learning curve, (theorizing again that a greater effect size may indicate that the model will converge more rapidly, and with a smaller sample size needed). Our results appear to indicate that this is not an effective heuristic for determining adequate sample size or projecting model performance, and therefore that additional work is still needed to better prospectively assess adequacy of data.