Efficient Online Scalar Annotation with Bounded Support

TL;DR

This paper introduces EASL, a hybrid method for efficiently collecting scalar annotations from humans, which improves accuracy and efficiency over traditional direct assessment and pairwise ranking methods.

Contribution

The paper proposes a novel hybrid approach, EASL, that combines direct assessment and pairwise ranking for better scalar annotation efficiency and accuracy.

Findings

EASL achieves higher correlation with ground truth.

EASL requires fewer annotator comparisons.

EASL improves dataset quality and system evaluation efficiency.

Abstract

We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.