Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

TL;DR

This paper investigates how BERT-based models generalize in natural language inference, analyzing various strategies and revealing insights into the limitations and potential of these models beyond simple heuristics.

Contribution

It provides a comprehensive case study of generalization strategies in NLI, comparing multiple architectures and techniques to understand their effectiveness.

Findings

Some strategies improve generalization to adversarial datasets

Certain approaches fail to enhance robustness

Insights into how Transformer models learn to generalize

Abstract

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.