Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop

TL;DR

This paper details the process of creating high-quality adversarial machine reading comprehension data through a collaborative approach involving humans and models, emphasizing annotation strategies, cost, and difficulty analysis.

Contribution

It introduces a novel data collection paradigm with humans and models in the loop, providing insights and recommendations for future adversarial data annotation efforts.

Findings

Successful adversarial attacks achieved

Cost analysis of annotation process conducted

Perceived difficulty varies with passage topics

Abstract

We present our experience as annotators in the creation of high-quality, adversarial machine-reading-comprehension data for extractive QA for Task 1 of the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an emergent data collection paradigm with both models and humans in the loop. We set up a quasi-experimental annotation design and perform quantitative analyses across groups with different numbers of annotators focusing on successful adversarial attacks, cost analysis, and annotator confidence correlation. We further perform a qualitative analysis of our perceived difficulty of the task given the different topics of the passages in our dataset and conclude with recommendations and suggestions that might be of value to people working on future DADC tasks and related annotation interfaces.