Computing Information Agreement

TL;DR

This paper extends the Information Agreement measure, an information-theoretic tool for evaluating diagnostic agreement, to handle agreement matrices with zero values, overcoming previous limitations.

Contribution

It introduces a modification to the Information Agreement measure allowing zero values in agreement matrices, broadening its applicability.

Findings

Extended IA can now handle zero values in matrices

Maintains robustness compared to Cohen's Kappa

Improves evaluation of diagnostic systems

Abstract

Agreement measures are useful to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. Information Agreement (IA) is an information-theoretic-based agreement measure introduced to overcome all the limitations and alleged pitfalls of Cohen's Kappa. However, it is only able to deal with agreement matrices whose values are positive natural numbers. This work extends IA admitting also 0 as a possible value for the agreement matrix cells.