It's easy to fool yourself: Case studies on identifying bias and confounding in bio-medical datasets

TL;DR

This paper presents two case studies demonstrating how biases and confounders in biomedical datasets can lead to misleading machine learning results, highlighting the importance of careful data analysis.

Contribution

It introduces practical examples of hidden biases in biomedical data and emphasizes the need for rigorous validation beyond model performance.

Findings

Biases from systematic data errors were identified.

Spurious signals unrelated to the prediction task were discovered.

High model performance can mask underlying confounders.

Abstract

Confounding variables are a well known source of nuisance in biomedical studies. They present an even greater challenge when we combine them with black-box machine learning techniques that operate on raw data. This work presents two case studies. In one, we discovered biases arising from systematic errors in the data generation process. In the other, we found a spurious source of signal unrelated to the prediction task at hand. In both cases, our prediction models performed well but under careful examination hidden confounders and biases were revealed. These are cautionary tales on the limits of using machine learning techniques on raw data from scientific experiments.