Estimators of different delta coefficients based on the unbiased estimator of the expected proportions of agreements

TL;DR

This paper introduces new, less biased estimators for delta coefficients used in measuring inter-rater agreement, especially effective with small sample sizes or few categories, and analyzes their statistical properties.

Contribution

It proposes novel estimators for delta coefficients and related parameters, improving bias and variance over existing methods, with applications including gold standard scenarios.

Findings

New estimators have lower bias than traditional ones.

Variance formulas for the estimators are derived.

Recommended use of new estimators when sample size or categories are small.

Abstract

To measure the degree of agreement between two observers that independently classify $n$ subjects within $K$ categories, it is common to use different kappa type coefficients, the most common of which is the $\kappa_C$ coefficient (Cohen's kappa). As $\kappa_C$ has some weaknesses -such as its poor performance with highly unbalanced marginal distributions-, the $\Delta$ coefficient is sometimes used, based on the $delta$ response model. This model allows us to obtain other parameters like: (a) the $\alpha_i$ contribution of each $i$ category to the value of the global agreement $\Delta=\sum \alpha_i$; and (b) the consistency $\mathcal{S}_i$ in the category $i$ (degree of agreement in the category $i$), a more appropriate parameter than the kappa value obtained by collapsing the data into the category $i$. It has recently been shown that the classic estimator $\hat{\kappa}_C$ underestimates $\kappa_C$, having obtained a new estimator $\hat{\kappa}_{CU}$ which is less biased. This article demonstrates that something similar happens to the known estimators $\hat{\Delta}$, $\hat{\alpha}_i$, and $\hat{\mathcal{S}}_i$ of $\Delta$, $\alpha_i$ and $\mathcal{S}_i$ (respectively), proposes new and less biased estimators $\hat{\Delta}_U$, $\hat{\alpha}_{iU}$, and $\hat{\mathcal{S}}_{iU}$, determines their variances, analyses the behaviour of all estimators, and concludes that the new estimators should be used when $n$ or $K$ are small (at least when $n\leq 50$ or $K\leq 3$). Additionally, the case where one of the raters is a gold standard is contemplated, in which situation two new parameters arise: the $conformity$ (the rater's capability to recognize a subject in the category $i$) and the $predictivity$ (the reliability of a response $i$ by the rater).