Methodological concerns about 'concordance-statistic for benefit' as a measure of discrimination in treatment benefit prediction

TL;DR

This paper critically examines the 'concordance statistic for benefit' (cfb) as a measure of discrimination in treatment benefit prediction, highlighting its limitations and proposing alternatives based on statistical dispersion.

Contribution

It demonstrates that cfb is not a proper scoring rule and is sensitive to unestimable correlations and pair definitions, suggesting alternative metrics.

Findings

cfb is not a proper scoring rule

cfb is sensitive to unestimable correlations

dispersion-based measures are viable alternatives

Abstract

Prediction algorithms that quantify the expected benefit of a given treatment conditional on patient characteristics can critically inform medical decisions. Quantifying the performance of treatment benefit prediction algorithms is an active area of research. A recently proposed metric, the concordance statistic for benefit (cfb), evaluates the discriminative ability of a treatment benefit predictor by directly extending the concept of the concordance statistic from a risk model with a binary outcome to a model for treatment benefit. In this work, we scrutinize $cfb$ on multiple fronts. Through numerical examples and theoretical developments, we show that cfb is not a proper scoring rule. We also show that it is sensitive to the unestimable correlation between counterfactual outcomes and to the definition of matched pairs. We argue that measures of statistical dispersion applied to predicted benefits do not suffer from these issues and can be an alternative metric for the discriminatory performance of treatment benefit predictors.