Does Putting a Linguist in the Loop Improve NLU Data Collection?

TL;DR

Involving linguists during crowdsourced NLU data collection helps create more challenging datasets and allows for dynamic gap mitigation, but does not necessarily improve out-of-domain model performance.

Contribution

This study demonstrates that real-time expert involvement during data collection enhances dataset quality and challenge level, introducing a novel iterative, linguist-in-the-loop protocol.

Findings

Linguist-in-the-loop datasets are more reliably challenging.

No significant improvement in out-of-domain performance with linguist involvement.

Chatroom interaction between linguists and crowdworkers has no measurable effect.

Abstract

Many crowdsourced NLP datasets contain systematic gaps and biases that are identified only after data collection is complete. Identifying these issues from early data samples during crowdsourcing should make mitigation more efficient, especially when done iteratively. We take natural language inference as a test case and ask whether it is beneficial to put a linguist `in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task. We directly compare three data collection protocols: (i) a baseline protocol, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints on the task, and (iii) an extension of linguist-in-the-loop that provides direct interaction between linguists and crowdworkers via a chatroom. The datasets collected with linguist involvement are more reliably challenging than baseline, without loss of quality. But we see no evidence that using this data in training leads to better out-of-domain model performance, and the addition of a chat platform has no measurable effect on the resulting dataset. We suggest integrating expert analysis \textit{during} data collection so that the expert can dynamically address gaps and biases in the dataset.