Assessing method agreement for paired repeated binary measurements administered by multiple raters

TL;DR

This paper introduces a model-based framework using generalized linear mixed models to assess agreement between two measurement methods and raters for repeated binary data, with applications in clinical research.

Contribution

It develops a unified approach to evaluate method and rater agreement in repeated binary measurements using GLMMs, including new assessment tools like Bland-Altman and Cohen's kappa adaptations.

Findings

Effective assessment of method agreement demonstrated through simulations.

Real clinical data analysis confirms the approach's practical utility.

The method reliably distinguishes interchangeable measurement methods.

Abstract

Method comparison studies are essential for development in medical and clinical fields. These studies often compare a cheaper, faster, or less invasive measuring method with a widely used one to see if they have sufficient agreement for interchangeable use. In the clinical and medical context, the response measurement is usually impacted not only by the measuring method but by the rater as well. This paper proposes a model-based approach to assess agreement of two measuring methods for paired repeated binary measurements under the scenario when the agreement between two measuring methods and the agreement among raters are required to be studied in a unified framework. Based upon the generalized linear mixed models (GLMM), the decision on the adequacy of interchangeable use is made by testing the equality of fixed effects of methods. Approaches for assessing method agreement, such as the Bland-Altman diagram and Cohen's kappa, are also developed for repeated binary measurements based upon the latent variables in GLMMs. We assess our novel model-based approach by simulation studies and a real clinical research application, in which patients are evaluated repeatedly for delirium with two validated screening methods: the Confusion Assessment Method and the 3-Minute Diagnostic Interview for Confusion Assessment Method. Both the simulation studies and the real data analyses demonstrate that our new approach can effectively assess method agreement.