Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology

TL;DR

This paper presents GCAM, a new annotation methodology that emphasizes guideline reporting, reduces information loss, and facilitates data reuse, thereby improving annotation quality and supporting better error analysis.

Contribution

GCAM introduces a prescriptive, guideline-centered annotation approach that enhances data quality and reusability compared to standard methods.

Findings

Improved annotation consistency and quality

Enhanced error analysis capabilities

Effective reuse of annotated data across tasks

Abstract

We introduce the Guideline-Centered Annotation Methodology (GCAM), a novel data annotation methodology designed to report the annotation guidelines associated with each data sample. Our approach addresses three key limitations of the standard prescriptive annotation methodology by reducing the information loss during annotation and ensuring adherence to guidelines. Furthermore, GCAM enables the efficient reuse of annotated data across multiple tasks. We evaluate GCAM in two ways: (i) through a human annotation study and (ii) an experimental evaluation with several machine learning models. Our results highlight the advantages of GCAM from multiple perspectives, demonstrating its potential to improve annotation quality and error analysis.