Partial-input baselines show that NLI models can ignore context, but they don't

TL;DR

This paper investigates whether state-of-the-art NLI models can effectively utilize context despite training on datasets with artifacts, demonstrating their capacity for contextual reasoning through a new evaluation set.

Contribution

The study introduces a perturbed premise evaluation set and shows that NLI models can learn to condition on context, challenging assumptions about reliance on spurious correlations.

Findings

NLI models can override partial-input baseline inferences.

Models demonstrate sensitivity to context perturbations.

Context conditioning is possible despite artifact-ridden training data.

Abstract

When strong partial-input baselines reveal artifacts in crowdsourced NLI datasets, the performance of full-input models trained on such datasets is often dismissed as reliance on spurious correlations. We investigate whether state-of-the-art NLI models are capable of overriding default inferences made by a partial-input baseline. We introduce an evaluation set of 600 examples consisting of perturbed premises to examine a RoBERTa model's sensitivity to edited contexts. Our results indicate that NLI models are still capable of learning to condition on context--a necessary component of inferential reasoning--despite being trained on artifact-ridden datasets.