Active Annotation: bootstrapping annotation lexicon and guidelines for supervised NLU learning

TL;DR

This paper introduces Active Annotation, a semi-automated approach combining unsupervised learning, human verification, and linguistic insights to efficiently create high-quality training data for NLU models, significantly reducing annotation time.

Contribution

It presents a novel Active Annotation framework that dynamically defines label spaces during annotation, improving efficiency and adaptability over traditional manual methods.

Findings

Achieves higher annotation quality with less effort.

Reduces annotation time by an order of magnitude.

Demonstrates effectiveness in a real NLU scenario.

Abstract

Natural Language Understanding (NLU) models are typically trained in a supervised learning framework. In the case of intent classification, the predicted labels are predefined and based on the designed annotation schema while the labelling process is based on a laborious task where annotators manually inspect each utterance and assign the corresponding label. We propose an Active Annotation (AA) approach where we combine an unsupervised learning method in the embedding space, a human-in-the-loop verification process, and linguistic insights to create lexicons that can be open categories and adapted over time. In particular, annotators define the y-label space on-the-fly during the annotation using an iterative process and without the need for prior knowledge about the input data. We evaluate the proposed annotation paradigm in a real use-case NLU scenario. Results show that our Active Annotation paradigm achieves accurate and higher quality training data, with an annotation speed of an order of magnitude higher with respect to the traditional human-only driven baseline annotation methodology.

Full text