Asymptotic confidence intervals for the difference and the ratio of the weighted kappa coefficients of two diagnostic tests subject to a paired design

TL;DR

This paper develops and evaluates asymptotic confidence intervals for comparing the weighted kappa coefficients of two diagnostic tests in paired studies, including methods for sample size calculation and practical application.

Contribution

It introduces new asymptotic confidence intervals for the difference and ratio of weighted kappa coefficients, along with a sample size calculation method and an R implementation.

Findings

Simulation shows good coverage probabilities of the proposed intervals.

The intervals have practical average lengths for real diagnostic data.

Application to malaria diagnosis demonstrates real-world utility.

Abstract

The weighted kappa coefficient of a binary diagnostic test is a measure of the beyond-chance agreement between the diagnostic test and the gold standard, and depends on the sensitivity and specificity of the diagnostic test, on the disease prevalence and on the relative importance between the false positives and the false negatives. This article studies the comparison of the weighted kappa coefficients of two binary diagnostic tests subject to a paired design through confidence intervals. Three asymptotic confidence intervals are studied for the difference between the parameters and five other intervals for the ratio. Simulation experiments were carried out to study the coverage probabilities and the average lengths of the intervals, giving some general rules for application. A method is also proposed to calculate the sample size necessary to compare the two weighted kappa coefficients through a confidence interval. A program in R has been written to solve the problem studied and it is available as supplementary material. The results were applied to a real example of the diagnosis of malaria.