No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

TL;DR

This paper introduces a novel no-gold-standard evaluation method for quantitative imaging that models correlated noise, enabling accurate ranking of imaging methods without needing a ground truth, especially in realistic noisy conditions.

Contribution

It proposes a maximum-likelihood-based NGSE technique that accounts for correlated noise, improving the evaluation and ranking of quantitative imaging methods without a gold standard.

Findings

Reliable estimation of noise-to-slope ratio (NSR) in 83% of trials.

Accurate identification of the most precise method in 97% of trials.

Effective ranking of imaging methods with correlated noise present.

Abstract

Objective evaluation of quantitative imaging (QI) methods with patient data is highly desirable, but is hindered by the lack or unreliability of an available gold standard. To address this issue, techniques that can evaluate QI methods without access to a gold standard are being actively developed. These techniques assume that the true and measured values are linearly related by a slope, bias, and Gaussian-distributed noise term, where the noise between measurements made by different methods is independent of each other. However, this noise arises in the process of measuring the same quantitative value, and thus can be correlated. To address this limitation, we propose a no-gold-standard evaluation (NGSE) technique that models this correlated noise by a multi-variate Gaussian distribution parameterized by a covariance matrix. We derive a maximum-likelihood-based approach to estimate the parameters that describe the relationship between the true and measured values, without any knowledge of the true values. We then use the estimated slopes and diagonal elements of the covariance matrix to compute the noise-to-slope ratio (NSR) to rank the QI methods on the basis of precision. The proposed NGSE technique was evaluated with multiple numerical experiments. Our results showed that the technique reliably estimated the NSR values and yielded accurate rankings of the considered methods for ~ 83% of 160 trials. In particular, the technique correctly identified the most precise method for ~ 97% of the trials. Overall, this study demonstrates the efficacy of the NGSE technique to accurately rank different QI methods when the correlated noise is present, and without access to any knowledge of the ground truth. The results motivate further validation of this technique with realistic simulation studies and patient data.