Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks

TL;DR

This paper investigates how generalized quantifiers impact the performance of multilingual NLU models, revealing their prevalence in benchmarks and the models' struggles with quantifier reasoning, especially in adversarial settings.

Contribution

It introduces an adversarial generalized quantifier NLI task (GQNLI) and demonstrates that current pre-trained models lack robustness in quantifier reasoning across languages.

Findings

Quantifiers are common in NLU benchmarks and cause performance drops.

Multilingual models show poor reasoning abilities with quantifiers, not necessarily worse than English.

Pre-trained models are vulnerable in adversarial quantifier reasoning tasks.

Abstract

Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.