Expected value of sample information calculations for risk prediction model development

TL;DR

This paper introduces a decision-theoretic approach to quantify the expected utility gain from additional data in risk prediction model development, emphasizing clinical utility over traditional statistical metrics.

Contribution

It proposes a bootstrap-based algorithm to compute the Expected Value of Sample Information (EVSI) for risk models, integrating utility considerations into study design.

Findings

EVSI can quantify utility gains from more data.

Bootstrap method is feasible for EVSI calculation.

Decision-theoretic metrics complement classical methods.

Abstract

Risk prediction models are often advertised as deterministic functions that map covariates to predicted risks. However, they are typically trained using finite samples, and as such, their predictions are inherently uncertain. This uncertainty has been addressed in terms of uncertainty around metrics of model performance (e.g., confidence intervals around c-statistic), as well as uncertainty or instability of predictions. Correspondingly, sample size calculations for model development studies target the precision of estimates of summary statistics and the stability of predictions. However, when evaluating the clinical utility of a model (as in Net Benefit (NB) calculations in decision curve analysis), statistical inference is less relevant. From a decision-theoretic perspective, the finite size of the sample results in utility loss due to the discrepancy between the fitted model and the correct model. From this perspective, procuring more development data is associated with an expected gain in the utility of using the model. In this work, we define the Expected Value of Sample Information (EVSI) as the expected gain in clinical utility, defined in NB terms, by procuring an additional development sample of a given size. We propose a bootstrap-based algorithm for EVSI computations and demonstrate its feasibility and face validity in a case study. We conclude that decision-theoretic metrics can complement classical inferential methods when designing studies aimed at developing risk prediction models.