An Evaluation Toolkit to Guide Model Selection and Cohort Definition in Causal Inference

TL;DR

This paper introduces a comprehensive evaluation toolkit for causal inference models using observational data, enabling better model selection, validation, and personalization in real-world healthcare datasets.

Contribution

The authors develop a modular, causal-specific evaluation toolkit that integrates with machine learning methods and demonstrates its use on a large rheumatoid arthritis dataset.

Findings

The toolkit improves model selection and data refinement for causal inference.

It enhances reproducibility and personalization in causal effect estimation.

Application on rheumatoid arthritis data shows practical utility.

Abstract

Real world observational data, together with causal inference, allow the estimation of causal effects when randomized controlled trials are not available. To be accepted into practice, such predictive models must be validated for the dataset at hand, and thus require a comprehensive evaluation toolkit, as introduced here. Since effect estimation cannot be evaluated directly, we turn to evaluating the various observable properties of causal inference, namely the observed outcome and treatment assignment. We developed a toolkit that expands established machine learning evaluation methods and adds several causal-specific ones. Evaluations can be applied in cross-validation, in a train-test scheme, or on the training data. Multiple causal inference methods are implemented within the toolkit in a way that allows modular use of the underlying machine learning models. Thus, the toolkit is agnostic to the machine learning model that is used. We showcase our approach using a rheumatoid arthritis cohort (consisting of about 120K patients) extracted from the IBM MarketScan(R) Research Database. We introduce an iterative pipeline of data definition, model definition, and model evaluation. Using this pipeline, we demonstrate how each of the evaluation components helps drive model selection and refinement of data extraction criteria in a way that provides more reproducible results and ensures that the causal question is answerable with available data. Furthermore, we show how the evaluation toolkit can be used to ensure that performance is maintained when applied to subsets of the data, thus allowing exploration of questions that move towards personalized medicine.