Using permutations to assess confounding in machine learning applications for digital health

TL;DR

This paper introduces permutation-based statistical methods to detect, quantify, and evaluate confounding effects in machine learning models applied to digital health data, enhancing understanding of model robustness without relying on modeling assumptions.

Contribution

The authors develop novel permutation-based tools for assessing confounding influence and unconfounded model performance, providing a robust alternative to existing adjustment methods.

Findings

Effective detection of confounding influence in digital health data

Quantification of unconfounded model performance

Application demonstrated on Parkinson's disease mobile health data

Abstract

Clinical machine learning applications are often plagued with confounders that can impact the generalizability and predictive performance of the learners. Confounding is especially problematic in remote digital health studies where the participants self-select to enter the study, thereby making it challenging to balance the demographic characteristics of participants. One effective approach to combat confounding is to match samples with respect to the confounding variables in order to balance the data. This procedure, however, leads to smaller datasets and hence impact the inferences drawn from the learners. Alternatively, confounding adjustment methods that make more efficient use of the data (e.g., inverse probability weighting) usually rely on modeling assumptions, and it is unclear how robust these methods are to violations of these assumptions. Here, rather than proposing a new approach to control for confounding, we develop novel permutation based statistical methods to detect and quantify the influence of observed confounders, and estimate the unconfounded performance of the learner. Our tools can be used to evaluate the effectiveness of existing confounding adjustment methods. We illustrate their application using real-life data from a Parkinson's disease mobile health study collected in an uncontrolled environment.