In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers

TL;DR

This paper introduces a three-stage workflow to identify, resolve, and clarify ambiguous instructions in crowdsourced annotation tasks, leading to improved accuracy and clearer guidelines.

Contribution

It presents a novel FIND-RESOLVE-LABEL workflow with design comparisons, enhancing annotation quality by systematically reducing ambiguity.

Findings

Improved annotation accuracy with the new workflow.

Collaborative FIND stage yields better ambiguous example detection.

Using both examples and tags enhances instruction clarity.

Abstract

We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced annotation to reduce ambiguity in task instructions and thus improve annotation quality. Stage 1 (FIND) asks the crowd to find examples whose correct label seems ambiguous given task instructions. Workers are also asked to provide a short tag which describes the ambiguous concept embodied by the specific instance found. We compare collaborative vs. non-collaborative designs for this stage. In Stage 2 (RESOLVE), the requester selects one or more of these ambiguous examples to label (resolving ambiguity). The new label(s) are automatically injected back into task instructions in order to improve clarity. Finally, in Stage 3 (LABEL), workers perform the actual annotation using the revised guidelines with clarifying examples. We compare three designs for using these examples: examples only, tags only, or both. We report image labeling experiments over six task designs using Amazon's Mechanical Turk. Results show improved annotation accuracy and further insights regarding effective design for crowdsourced annotation tasks.