Dodgersort: Uncertainty-Aware VLM-Guided Human-in-the-Loop Pairwise Ranking

TL;DR

Dodgersort is a novel uncertainty-aware ranking method that reduces human annotation effort and enhances reliability in visual ranking tasks across domains by combining hierarchical pre-ordering, neural ranking, and probabilistic ensembles.

Contribution

It introduces Dodgersort, integrating CLIP-based pre-ordering, neural ranking, and uncertainty modeling to improve efficiency and reliability in human-in-the-loop pairwise ranking.

Findings

Achieves 11-16% annotation reduction in medical imaging, dating, and aesthetics.

Extracts 5-20 times more ranking information per comparison in age estimation.

Improves inter-rater reliability across multiple datasets.

Abstract

Pairwise comparison labeling is emerging as it yields higher inter-rater reliability than conventional classification labeling, but exhaustive comparisons require quadratic cost. We propose Dodgersort, which leverages CLIP-based hierarchical pre-ordering, a neural ranking head and probabilistic ensemble (Elo, BTL, GP), epistemic--aleatoric uncertainty decomposition, and information-theoretic pair selection. It reduces human comparisons while improving the reliability of the rankings. In visual ranking tasks in medical imaging, historical dating, and aesthetics, Dodgersort achieves a 11--16\% annotation reduction while improving inter-rater reliability. Cross-domain ablations across four datasets show that neural adaptation and ensemble uncertainty are key to this gain. In FG-NET with ground-truth ages, the framework extracts 5--20$\times$ more ranking information per comparison than baselines, yielding Pareto-optimal accuracy--efficiency trade-offs.