Adjusted Similarity Measures and a Violation of Expectations

TL;DR

This paper investigates the properties of adjusted similarity measures used for comparing discrete labels, highlighting potential violations of their expected null properties under various null models and standardization procedures.

Contribution

It generalizes the adjustment operator to broader null models and provides conditions ensuring the measures retain their intended properties.

Findings

Violations can lead to measures with unexpected values or breakdown of null properties.

Traditional adjustments may not hold under alternative null models such as clustering ensembles.

Proper incorporation of observed data is crucial for maintaining measure validity.

Abstract

Adjusted similarity measures, such as Cohen's kappa for inter-rater reliability and the adjusted Rand index used to compare clustering algorithms, are a vital tool for comparing discrete labellings. These measures are intended to have the property of 0 expectation under a null distribution and maximum value 1 under maximal similarity to aid in interpretation. Measures are frequently adjusted with respect to the permutation distribution for historic and analytic reasons. There is currently renewed interest in considering other null models more appropriate for context, such as clustering ensembles permitting a random number of identified clusters. The purpose of this work is two -- fold: (1) to generalize the study of the adjustment operator to general null models and to a more general procedure which includes statistical standardization as a special case and (2) to identify sufficient conditions for the adjustment operator to produce the intended properties, where sufficient conditions are related to whether and how observed data are incorporated into null distributions. We demonstrate how violations of the sufficient conditions may lead to substantial breakdown, such as by producing a non-positive measure under traditional adjustment rather than one with mean 0, or by producing a measure which is deterministically 0 under statistical standardization.