Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory

TL;DR

This paper introduces a novel data partitioning method using Item Response Theory to improve ML model validation by accounting for instance quality, revealing data heterogeneity, and optimizing bias-variance tradeoffs.

Contribution

It applies IRT parameters to dataset partitioning, demonstrating improved understanding of data heterogeneity and model performance impacts in ML validation.

Findings

IRT reveals data heterogeneity and informative subgroups.

Balanced partitions improve bias-variance understanding.

High-guessing instances can impair model accuracy significantly.

Abstract

Robust validation of Machine Learning (ML) models is essential, but traditional data partitioning approaches often ignore the intrinsic quality of each instance. This study proposes the use of Item Response Theory (IRT) parameters to characterize and guide the partitioning of datasets in the model validation stage. The impact of IRT-informed partitioning strategies on the performance of several ML models in four tabular datasets was evaluated. The results obtained demonstrate that IRT reveals an inherent heterogeneity of the instances and highlights the existence of informative subgroups of instances within the same dataset. Based on IRT, balanced partitions were created that consistently help to better understand the tradeoff between bias and variance of the models. In addition, the guessing parameter proved to be a determining factor: training with high-guessing instances can significantly impair model performance and resulted in cases with accuracy below 50%, while other partitions reached more than 70% in the same dataset.