Three-way Mixed Effect ANOVA to Estimate MRMC Limits of Agreement

TL;DR

This paper introduces a three-way mixed effect ANOVA model to estimate limits of agreement in multi-reader multi-case studies, validated through simulation to ensure unbiased variance estimates and assess uncertainty.

Contribution

It presents a novel application of three-way mixed effect ANOVA for MRMC agreement analysis, including a hierarchical simulation model for validation.

Findings

Variance estimates are unbiased.

Uncertainty decreases with more readers and cases.

Variance increases with higher true variance.

Abstract

When evaluating the clinical performance of a medical imaging device, a multi-reader multi-case (MRMC) analysis is usually applied to account for both case and reader variability. For a clinical task that equates to a quantitative measurement, an agreement analysis such as a limits of agreement (LOA) method can be used to compare different measurement methods. In this work, we introduce four types of comparisons; these types differ depending on whether the measurements are within or between readers and within or between modalities. A three-way mixed effect ANOVA model is applied to estimate the variances of individual differences, which is an essential step for estimating LOA. To verify the estimates of LOA, we propose a hierarchical model to simulate quantitative MRMC data. Two simulation studies were conducted to validate both the simulation and the LOA variance estimates. From the simulation results, we can conclude that our estimate of variance is unbiased, and the uncertainty of the estimation drops as the number of readers and cases increases and rises as the value of true variance increases.