InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

TL;DR

This paper introduces InferES, a high-quality Spanish NLI corpus with negation-based adversarial examples, and evaluates transformer models' performance and generalization capabilities on this dataset.

Contribution

The creation of InferES, a novel Spanish NLI corpus with focus on negation and adversarial examples, and analysis of model performance and generalization on this data.

Findings

Transformer models achieved 72.8% accuracy on InferES.

Models trained on InferES generalize well across topics.

Negation-based adversarial examples pose moderate challenges for models.

Abstract

In this paper, we present InferES - an original corpus for Natural Language Inference (NLI) in European Spanish. We propose, implement, and analyze a variety of corpus-creating strategies utilizing expert linguists and crowd workers. The objectives behind InferES are to provide high-quality data, and, at the same time to facilitate the systematic evaluation of automated systems. Specifically, we focus on measuring and improving the performance of machine learning systems on negation-based adversarial examples and their ability to generalize across out-of-distribution topics. We train two transformer models on InferES (8,055 gold examples) in a variety of scenarios. Our best model obtains 72.8% accuracy, leaving a lot of room for improvement. The "hypothesis-only" baseline performs only 2%-5% higher than majority, indicating much fewer annotation artifacts than prior work. We find that models trained on InferES generalize very well across topics (both in- and out-of-distribution) and perform moderately well on negation-based adversarial examples.