Estimating the Causal Effects of Natural Logic Features in Neural NLI Models

TL;DR

This paper investigates the causal effects of semantic features in neural NLI models by applying causal effect estimation to understand reasoning failures and model robustness in natural language inference tasks.

Contribution

It introduces a causal analysis framework for NLI models focusing on semantic monotonicity and word relations, providing insights into model reasoning and robustness.

Findings

Models show systematic reasoning failures in semantic monotonicity.

Interventions on word pairs significantly affect entailment predictions.

The approach enables comparative profiling of model robustness.

Abstract

Rigorous evaluation of the causal effects of semantic features on language model predictions can be hard to achieve for natural language reasoning problems. However, this is such a desirable form of analysis from both an interpretability and model evaluation perspective, that it is valuable to zone in on specific patterns of reasoning with enough structure and regularity to be able to identify and quantify systematic reasoning failures in widely-used models. In this vein, we pick a portion of the NLI task for which an explicit causal diagram can be systematically constructed: in particular, the case where across two sentences (the premise and hypothesis), two related words/terms occur in a shared context. In this work, we apply causal effect estimation strategies to measure the effect of context interventions (whose effect on the entailment label is mediated by the semantic monotonicity characteristic) and interventions on the inserted word-pair (whose effect on the entailment label is mediated by the relation between these words.). Following related work on causal analysis of NLP models in different settings, we adapt the methodology for the NLI task to construct comparative model profiles in terms of robustness to irrelevant changes and sensitivity to impactful changes.