K$\alpha$LOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks

TL;DR

KαLOS is a unified, data-driven meta-algorithm that standardizes dataset quality evaluation in complex vision tasks by resolving spatial correspondence and calibrating agreement metrics, improving reliability and diagnostics.

Contribution

It introduces KαLOS, a principled framework that generalizes agreement evaluation across diverse vision tasks, addressing the limitations of existing heuristics and validation methods.

Findings

KαLOS effectively calibrates localization parameters to inherent agreement distributions.

The framework enables granular diagnostics like annotator vitality and localization sensitivity.

Empirical validation with a novel noise generator demonstrates robustness in distinguishing signal from noise.

Abstract

Progress in object detection benchmarks is stagnating. It is limited not by architectures but by the inability to distinguish model improvements from label noise. To restore trust in benchmarking the field requires rigorous quantification of annotation consistency to ensure the reliability of evaluation data. However, standard statistical metrics fail to handle the instance correspondence problem inherent to vision tasks. Furthermore, validating new agreement metrics remains circular because no objective ground truth for agreement exists. This forces reliance on unverifiable heuristics. We propose K$\alpha$LOS (KALOS), a unified meta-algorithm that generalizes the "Localization First" principle to standardize dataset quality evaluation. By resolving spatial correspondence before assessing agreement, our framework transforms complex spatio-categorical problems into nominal reliability matrices. Unlike prior heuristic implementations, K$\alpha$LOS employs a principled, data-driven configuration; by statistically calibrating the localization parameters to the inherent agreement distribution, it generalizes to diverse tasks ranging from bounding boxes to volumetric segmentation or pose estimation. This standardization enables granular diagnostics beyond a single score. These include annotator vitality, collaboration clustering, and localization sensitivity. To validate this approach, we introduce a novel and empirically derived noise generator. Where prior validations relied on uniform error assumptions, our controllable testbed models complex and non-isotropic human variability. This provides evidence of the metric's properties and establishes K$\alpha$LOS as a robust standard for distinguishing signal from noise in modern computer vision benchmarks.