Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model

TL;DR

This paper introduces SANT, a framework that optimizes data annotation by intelligently allocating data to experts or models, improving annotation quality within budget constraints.

Contribution

The paper proposes a novel selective annotation framework, SANT, that effectively allocates data to experts or models using error-aware triage and bi-weighting mechanisms.

Findings

SANT outperforms baseline methods in annotation quality.

Efficient data allocation reduces unnecessary expert annotation.

Experimental results validate the effectiveness of the proposed approach.

Abstract

To obtain high-quality annotations under limited budget, semi-automatic annotation methods are commonly used, where a portion of the data is annotated by experts and a model is then trained to complete the annotations for the remaining data. However, these methods mainly focus on selecting informative data for expert annotations to improve the model predictive ability (i.e., triage-to-human data), while the rest of the data is indiscriminately assigned to model annotation (i.e., triage-to-model data). This may lead to inefficiencies in budget allocation for annotations, as easy data that the model could accurately annotate may be unnecessarily assigned to the expert, and hard data may be misclassified by the model. As a result, the overall annotation quality may be compromised. To address this issue, we propose a selective annotation framework called SANT. It effectively takes advantage of both the triage-to-human and triage-to-model data through the proposed error-aware triage and bi-weighting mechanisms. As such, informative or hard data is assigned to the expert for annotation, while easy data is handled by the model. Experimental results show that SANT consistently outperforms other baselines, leading to higher-quality annotation through its proper allocation of data to both expert and model workers. We provide pioneering work on data annotation within budget constraints, establishing a landmark for future triage-based annotation studies.