Truth Discovery in Sequence Labels from Crowds

TL;DR

This paper introduces AggSLC, an optimization-based method for accurately inferring true sequence labels from crowdsourced annotations, effectively handling dependencies and worker reliability in NLP tasks.

Contribution

It proposes a novel aggregation algorithm for sequential labels that considers task characteristics, worker reliability, and machine learning, with proven convergence and superior performance.

Findings

Outperforms existing aggregation methods on NER and biomedical datasets.

Demonstrates effective handling of sequential dependencies and worker reliability.

Shows convergence of the proposed algorithm after finite iterations.

Abstract

Annotation quality and quantity positively affect the learning performance of sequence labeling, a vital task in Natural Language Processing. Hiring domain experts to annotate a corpus is very costly in terms of money and time. Crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), have been deployed to assist in this purpose. However, the annotations collected this way are prone to human errors due to the lack of expertise of the crowd workers. Existing literature in annotation aggregation assumes that annotations are independent and thus faces challenges when handling the sequential label aggregation tasks with complex dependencies. To conquer the challenges, we propose an optimization-based method that infers the ground truth labels using annotations provided by workers for sequential labeling tasks. The proposed Aggregation method for Sequential Labels from Crowds ($AggSLC$) jointly considers the characteristics of sequential labeling tasks, workers' reliabilities, and advanced machine learning techniques. Theoretical analysis on the algorithm's convergence further demonstrates that the proposed $AggSLC$ halts after a finite number of iterations. We evaluate $AggSLC$ on different crowdsourced datasets for Named Entity Recognition (NER) tasks and Information Extraction tasks in biomedical (PICO), as well as a simulated dataset. Our results show that the proposed method outperforms the state-of-the-art aggregation methods. To achieve insights into the framework, we study the effectiveness of $AggSLC$'s components through ablation studies.