QuMAB: Query-based Multi-Annotator Behavior Modeling with Reliability under Sparse Labels

TL;DR

QuMAB introduces a novel approach to multi-annotator learning by modeling individual annotator behavior patterns using lightweight queries, improving reliability and interpretability especially with sparse labels.

Contribution

It shifts from sample-wise aggregation to annotator-wise behavior modeling, proposing QuMAB with query-based modeling and new large-scale datasets for multi-annotator analysis.

Findings

QuMAB outperforms existing methods in modeling annotator behavior.

It enhances consensus prediction accuracy.

The approach is effective under sparse annotation conditions.

Abstract

Multi-annotator learning traditionally aggregates diverse annotations to approximate a single ground truth, treating disagreements as noise. However, this paradigm faces fundamental challenges: subjective tasks often lack absolute ground truth, and sparse annotation coverage makes aggregation statistically unreliable. We introduce a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling. By treating annotator disagreements as valuable information rather than noise, modeling annotator-specific behavior patterns can reconstruct unlabeled data to reduce annotation cost, enhance aggregation reliability, and explain annotator decision behavior. To this end, we propose QuMAB (Query-based Multi-Annotator Behavior Pattern Learning), which uses light-weight queries to model individual annotators while capturing inter-annotator correlations as implicit regularization, preventing overfitting to sparse individual data while maintaining individualization and improving generalization, with a visualization of annotator focus regions offering an explainable analysis of behavior understanding. We contribute two large-scale datasets with dense per-annotator labels: STREET (4,300 labels/annotator) and AMER (average 3,118 labels/annotator), the first multimodal multi-annotator dataset. Extensive experiments demonstrate the superiority of our QuMAB in modeling individual annotators' behavior patterns, their utility for consensus prediction, and applicability under sparse annotations.