Revisiting Active Learning under (Human) Label Variation

TL;DR

This paper explores the impact of human label variation on active learning, proposing a framework to incorporate plausible label differences and improve annotation strategies in real-world scenarios.

Contribution

It introduces a conceptual framework for integrating human label variation into active learning processes, addressing overlooked complexities in label quality and annotation practices.

Findings

Decomposition of label variation into signal and noise.

Survey of existing approaches to label variation and active learning.

Proposal of a HLV-aware active learning framework.

Abstract

Access to high-quality labeled data remains a limiting factor in applied supervised learning. While label variation (LV), i.e., differing labels for the same instance, is common, especially in natural language processing, annotation frameworks often still rest on the assumption of a single ground truth. This overlooks human label variation (HLV), the occurrence of plausible differences in annotations, as an informative signal. Similarly, active learning (AL), a popular approach to optimizing the use of limited annotation budgets in training ML models, often relies on at least one of several simplifying assumptions, which rarely hold in practice when acknowledging HLV. In this paper, we examine foundational assumptions about truth and label nature, highlighting the need to decompose observed LV into signal (e.g., HLV) and noise (e.g., annotation error). We survey how the AL and (H)LV communities have addressed -- or neglected -- these distinctions and propose a conceptual framework for incorporating HLV throughout the AL loop, including instance selection, annotator choice, and label representation. We further discuss the integration of large language models (LLM) as annotators. Our work aims to lay a conceptual foundation for HLV-aware active learning, better reflecting the complexities of real-world annotation.