Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models

TL;DR

This paper investigates how cognitive heuristics used by annotators influence data quality and model robustness in reading comprehension datasets, revealing that heuristic use correlates with biases and impacts model performance.

Contribution

It introduces a method to track annotator heuristic traces and demonstrates their effect on data bias and model generalization in NLP tasks.

Findings

Annotators use multiple cognitive heuristics during annotation.

Heuristic use correlates with increased data bias and model susceptibility.

Tracking heuristics can improve dataset quality and bias diagnosis.

Abstract

Cognitive psychologists have documented that humans use cognitive heuristics, or mental shortcuts, to make quick decisions while expending less effort. While performing annotation work on crowdsourcing platforms, we hypothesize that such heuristic use among annotators cascades on to data quality and model robustness. In this work, we study cognitive heuristic use in the context of annotating multiple-choice reading comprehension datasets. We propose tracking annotator heuristic traces, where we tangibly measure low-effort annotation strategies that could indicate usage of various cognitive heuristics. We find evidence that annotators might be using multiple such heuristics, based on correlations with a battery of psychological tests. Importantly, heuristic use among annotators determines data quality along several dimensions: (1) known biased models, such as partial input models, more easily solve examples authored by annotators that rate highly on heuristic use, (2) models trained on annotators scoring highly on heuristic use don't generalize as well, and (3) heuristic-seeking annotators tend to create qualitatively less challenging examples. Our findings suggest that tracking heuristic usage among annotators can potentially help with collecting challenging datasets and diagnosing model biases.