Comparing evaluation of quantitative imaging methods using reference standards vs. regression-without-truth-based technique

TL;DR

This study compares the effectiveness of reference standards versus regression-without-truth techniques for evaluating quantitative imaging methods, especially when reference standards are unreliable.

Contribution

It provides controlled experimental evidence showing when RWT techniques outperform reference standards for evaluation without gold standards.

Findings

RWT techniques outperform reference standards when measurement error is high.

Evaluation accuracy depends on the error level in reference standards.

The study offers guidance for choosing evaluation methods in clinical imaging.

Abstract

Clinical translation of quantitative imaging (QI) methods requires objective evaluation of these methods on reliably measuring the underlying true quantitative values. Ideally, such evaluation would be performed using ground truth or gold standards. However, in clinical practice, ground truth is generally unavailable, and obtaining gold standards for large patient cohorts is often impractical or infeasible. In these scenarios, evaluation using well-accepted or commonly used methods for measuring the quantitative values that may have bias and/or measurement error, referred to as reference standards, is a common alternative. However, the error in reference standards might lead to inaccurate results in evaluating the QI methods. To address the challenge of evaluation without a gold standard, a class of regression-without-truth (RWT)-based techniques has also been proposed. These techniques compare QI methods without requiring ground truth and have demonstrated promise across multiple applications. This raises an important question: how does evaluation using reference standards vs. RWT-based techniques compare? Our work, which investigates this question using controlled numerical experiments, showed that when the measurement error of the reference standard exceeded a certain level, evaluation with a RWT-based technique outperformed evaluation with reference standards. This study provides insights for selecting appropriate evaluation strategies when gold standards are unavailable and motivates further investigation in clinically realistic settings.