Bayesian Joint Modeling of Interrater and Intrarater Reliability with Multilevel Data

TL;DR

This paper introduces three Bayesian models for assessing interrater and intrarater reliability in multilevel data, providing new estimates and formulas, with applications to real datasets and simulations.

Contribution

The paper develops and implements three generalized Bayesian models for reliability analysis, including formulas for marginal correlations and comparative evaluations.

Findings

New Bayesian models for reliability estimation

Formulas for marginal correlations derived

Model comparisons on real datasets and simulations

Abstract

We formulate three generalized Bayesian models for analyzing interrater and intrarater reliability in the presence of multilevel data. Stan implementations of these models provide new estimates of interrater and intrarater reliability. We also derive formulas for calculating marginal correlations under each of the three models. Comparisons of the kappa estimates and marginal correlations across the different models are presented from two real-world datasets. Simulations demonstrate properties of the different measures of agreement under different model assumptions.