Decomposing Natural Logic Inferences in Neural NLI

TL;DR

This paper investigates whether neural NLI models effectively capture semantic features like monotonicity and concept inclusion, revealing that high-performing models often lack these crucial features in their representations.

Contribution

It systematically probes neural NLI models to assess their encoding of natural logic features and shows how fine-tuning enhances these semantic representations.

Findings

Monotonicity information is weak in popular NLI models' embeddings.

Fine-tuning improves models' monotonicity features and performance on challenge sets.

High benchmark scores do not necessarily indicate strong semantic feature capture.

Abstract

In the interest of interpreting neural NLI models and their reasoning strategies, we carry out a systematic probing study which investigates whether these models capture the crucial semantic features central to natural logic: monotonicity and concept inclusion. Correctly identifying valid inferences in downward-monotone contexts is a known stumbling block for NLI performance, subsuming linguistic phenomena such as negation scope and generalized quantifiers. To understand this difficulty, we emphasize monotonicity as a property of a context and examine the extent to which models capture monotonicity information in the contextual embeddings which are intermediate to their decision making process. Drawing on the recent advancement of the probing paradigm, we compare the presence of monotonicity features across various models. We find that monotonicity information is notably weak in the representations of popular NLI models which achieve high scores on benchmarks, and observe that previous improvements to these models based on fine-tuning strategies have introduced stronger monotonicity features together with their improved performance on challenge sets.