Statistically Profiling Biases in Natural Language Reasoning Datasets and Models

TL;DR

This paper introduces ICQ, a lightweight framework that automatically detects biases in natural language reasoning datasets and assesses how models exploit these biases, addressing limitations of manual stress tests.

Contribution

The paper presents ICQ, a novel statistical profiling method that identifies biases in datasets and evaluates model reliance without additional test case creation.

Findings

ICQ effectively detects biases in multiple-choice NLU datasets.

Models tend to exploit identified biases, overestimating their reasoning capabilities.

ICQ provides a scalable, general approach for bias detection in NLP datasets.

Abstract

Recent work has indicated that many natural language understanding and reasoning datasets contain statistical cues that may be taken advantaged of by NLP models whose capability may thus be grossly overestimated. To discover the potential weakness in the models, some human-designed stress tests have been proposed but they are expensive to create and do not generalize to arbitrary models. We propose a light-weight and general statistical profiling framework, ICQ (I-See-Cue), which automatically identifies possible biases in any multiple-choice NLU datasets without the need to create any additional test cases, and further evaluates through blackbox testing the extent to which models may exploit these biases.