On Variants of Root Normalised Order-aware Divergence and a Divergence based on Kendall's Tau

TL;DR

This study evaluates variants of Root Normalised Order-aware Divergence (RNOD) and introduces Divergence based on Kendall's Tau (DNKT) for ordinal quantification, finding RNOD most robust and DNKT less reliable in system ranking consistency.

Contribution

The paper proposes new variants of RNOD and introduces DNKT, analyzing their effectiveness and robustness in ordinal quantification tasks.

Findings

RNOD variants do not improve system ranking consistency over original RNOD

RNOD is the most robust measure among those tested

DNKT shows poor performance in system ranking stability

Abstract

This paper reports on a follow-up study of the work reported in Sakai, which explored suitable evaluation measures for ordinal quantification tasks. More specifically, the present study defines and evaluates, in addition to the quantification measures considered earlier, a few variants of an ordinal quantification measure called Root Normalised Order-aware Divergence (RNOD), as well as a measure which we call Divergence based on Kendall's $\tau$ (DNKT). The RNOD variants represent alternative design choices based on the idea of Sakai's Distance-Weighted sum of squares (DW), while DNKT is designed to ensure that the system's estimated distribution over classes is faithful to the target priorities over classes. As this Priority Preserving Property (PPP) of DNKT may be useful in some applications, we also consider combining some of the existing quantification measures with DNKT. Our experiments with eight ordinal quantification data sets suggest that the variants of RNOD do not offer any benefit over the original RNOD at least in terms of system ranking consistency, i.e., robustness of the system ranking to the choice of test data. Of all ordinal quantification measures considered in this study (including Normalised Match Distance, a.k.a. Earth Mover's Distance), RNOD is the most robust measure overall. Hence the design choice of RNOD is a good one from this viewpoint. Also, DNKT is the worst performer in terms of system ranking consistency. Hence, if DNKT seems appropriate for a task, sample size design should take its statistical instability into account.