Evaluating Predictive Modeling Strategies for Predicting Individual Treatment Effects in Precision Medicine

TL;DR

This study systematically compares over 30 modeling strategies for predicting individual treatment effects in precision medicine, emphasizing the importance of external validation and robust methods like penalized regressions.

Contribution

It provides a comprehensive evaluation of various modeling approaches for PITE estimation, highlighting the effectiveness of penalized and projection-based methods.

Findings

Penalized and projection-based methods consistently perform well.

External validation reveals model weaknesses not seen in internal validation.

Flexible learners require strong signals and large samples to excel.

Abstract

Precision medicine seeks to match patients with treatments that produce the greatest benefit. The Predicted Individual Treatment Effect (PITE)-the difference between predicted outcomes under treatment and control-quantifies this benefit but is difficult to estimate due to unobserved counterfactuals, high dimensionality, and complex interactions. We compared 30+ modeling strategies, including penalized and projection-based methods, flexible learners, and tree-ensembles, using a structured simulation framework varying sample size, dimensionality, multicollinearity, and interaction complexity. Performance was measured using root mean squared error (RMSE) for prediction accuracy and directional accuracy (DIR) for correctly classifying benefit versus harm. Internal validation produced optimistic estimates, whereas external validation with distributional shifts and higher-order interactions more clearly revealed model weaknesses. Penalized and projection-based approaches-ridge, lasso, elastic net, partial least squares (PLS), and principal components regression (PCR)-consistently achieved strong RMSE and DIR performance. Flexible learners excelled only under strong signals and sufficient sample sizes. Results highlight robust linear/projection defaults and the necessity of rigorous external validation.