Disparate Model Performance and Stability in Machine Learning Clinical Support for Diabetes and Heart Diseases

TL;DR

This study uncovers sex- and age-related disparities in machine learning models for chronic disease prediction, emphasizing the need to address data and model biases to ensure equitable clinical outcomes.

Contribution

Introduces a novel analytical framework combining traditional metrics with data complexity to evaluate model fairness across demographic groups.

Findings

Mild sex-related disparities favoring males.

Significant age-related differences with better accuracy for younger patients.

Older patients show inconsistent accuracy linked to data complexity.

Abstract

Machine Learning (ML) algorithms are vital for supporting clinical decision-making in biomedical informatics. However, their predictive performance can vary across demographic groups, often due to the underrepresentation of historically marginalized populations in training datasets. The investigation reveals widespread sex- and age-related inequities in chronic disease datasets and their derived ML models. Thus, a novel analytical framework is introduced, combining systematic arbitrariness with traditional metrics like accuracy and data complexity. The analysis of data from over 25,000 individuals with chronic diseases revealed mild sex-related disparities, favoring predictive accuracy for males, and significant age-related differences, with better accuracy for younger patients. Notably, older patients showed inconsistent predictive accuracy across seven datasets, linked to higher data complexity and lower model performance. This highlights that representativeness in training data alone does not guarantee equitable outcomes, and model arbitrariness must be addressed before deploying models in clinical settings.