Interpretation of machine learning predictions for patient outcomes in electronic health records

TL;DR

This study evaluates different methods for interpreting machine learning models predicting patient outcomes from electronic health records, highlighting the variability and reliability of feature importance measures.

Contribution

It systematically compares multiple feature importance techniques across models, revealing their inconsistencies and identifying permutation tests as the most clinically interpretable.

Findings

Permutation tests align best with clinical interpretation.

Feature importance methods often disagree despite similar model performance.

Poor correlation observed among different importance assessment techniques.

Abstract

Electronic health records are an increasingly important resource for understanding the interactions between patient health, environment, and clinical decisions. In this paper we report an empirical study of predictive modeling of several patient outcomes using three state-of-the-art machine learning methods. Our primary goal is to validate the models by interpreting the importance of predictors in the final models. Central to interpretation is the use of feature importance scores, which vary depending on the underlying methodology. In order to assess feature importance, we compared univariate statistical tests, information-theoretic measures, permutation testing, and normalized coefficients from multivariate logistic regression models. In general we found poor correlation between methods in their assessment of feature importance, even when their performance is comparable and relatively good. However, permutation tests applied to random forest and gradient boosting models showed the most agreement, and the importance scores matched the clinical interpretation most frequently.