Everything is Varied: The Surprising Impact of Individual Variation on ML Robustness in Medicine

TL;DR

This paper investigates how individual variation in medical data affects machine learning robustness, revealing that common models are vulnerable and proposing strategies to improve their resilience for clinical applications.

Contribution

It formalizes the problem of individual variation in ML and demonstrates effective mitigation strategies through experiments on real-world medical data.

Findings

State-of-the-art ML models are severely impacted by individual variation.

Data augmentation and imprecisiation improve robustness to individual variation.

Proper study design can mitigate the effects of individual variation.

Abstract

In medical settings, Individual Variation (IV) refers to variation that is due not to population differences or errors, but rather to within-subject variation, that is the intrinsic and characteristic patterns of variation pertaining to a given instance or the measurement process. While taking into account IV has been deemed critical for proper analysis of medical data, this source of uncertainty and its impact on robustness have so far been neglected in Machine Learning (ML). To fill this gap, we look at how IV affects ML performance and generalization and how its impact can be mitigated. Specifically, we provide a methodological contribution to formalize the problem of IV in the statistical learning framework and, through an experiment based on one of the largest real-world laboratory medicine datasets for the problem of COVID-19 diagnosis, we show that: 1) common state-of-the-art ML models are severely impacted by the presence of IV in data; and 2) advanced learning strategies, based on data augmentation and data imprecisiation, and proper study designs can be effective at improving robustness to IV. Our findings demonstrate the critical relevance of correctly accounting for IV to enable safe deployment of ML in clinical settings.