Overall Agreement for Multiple Raters with Replicated Measurements

TL;DR

This paper introduces new nonparametric agreement indices for multiple raters that preserve intuitive interpretation and accommodate replicated measurements, improving assessment of inter-rater reliability without distributional assumptions.

Contribution

It proposes a novel set of overall agreement indices based on maximum pairwise differences and a unified GEE-based estimation method that handles replications and maintains interpretability.

Findings

Proposed indices retain intuitive interpretation from pairwise agreement.

The GEE-based estimator achieves efficiency bounds under mild conditions.

Simulation studies demonstrate the method's effectiveness across scenarios.

Abstract

Multiple raters are often needed to be used interchangeably in practice for measurement or evaluation. Assessing agreement among these multiple raters via agreement indices are necessary before their participation. While the intuitively appealing agreement indices such as coverage probability and total deviation index, and relative area under coverage probability curve, have been extended for assessing overall agreement among multiple raters, these extensions have limitations. The existing overall agreement indices either require normality and homogeneity assumptions or did not preserve the intuitive interpretation of the indices originally defined for two raters. In this paper, we propose a new set of overall agreement indices based on maximum pairwise differences among all raters. The proposed new overall coverage probability, overall total deviation index and relative area under overall coverage probability curve retain the original intuitive interpretation from the pairwise version. Without making any distributional assumption, we also propose a new unified nonparametric estimation and inference approach for the overall indices based on generalized estimating equations that can accommodate replications made by the same rater. Under mild assumptions, the proposed variance estimator is shown to achieve efficiency bound under independent working correlation matrix. Simulation studies under different scenarios are conducted to assess the performance of the proposed estimation and inference approach with and without replications. We illustrate the methodology by using a blood pressure data with three raters who made three replications on each subjects.