The DALPHI annotation framework & how its pre-annotations can improve annotator efficiency

TL;DR

This paper introduces DALPHI, an annotation framework that uses active learning to provide pre-annotations, significantly improving annotation efficiency and quality in NLP tasks, demonstrated through a study on named entity annotation.

Contribution

The paper presents DALPHI, a novel annotation framework that leverages active learning to enhance annotation efficiency and quality with pre-annotations, even at 50% recall.

Findings

Pre-annotations improve annotation quality and quantity.

Active learning-based assistance reduces annotator effort.

Even with 50% recall, pre-annotations are beneficial.

Abstract

Producing the required amounts of training data for machine learning and NLP tasks often involves human annotators doing very repetitive and monotonous work. In this paper, we present and evaluate our novel annotation framework DALPHI, which facilitates the annotation process by providing the annotator with suggestions generated by an automated, active-learning based assistance system. In a study with 66 participants, we demonstrate on the exemplary task of annotating named entities in text documents that with this assistance system the annotation processes can be improved with respect to the quality and quantity of produced annotations, even if the pre-annotations provided by the assistance system are at a recall level of only 50%.