Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models

TL;DR

This paper introduces a logical reasoning framework for NLI that enhances interpretability and robustness without sacrificing accuracy, by identifying hypothesis spans responsible for predictions and leveraging human explanations during training.

Contribution

It presents a novel span-level prediction approach for NLI that improves interpretability and robustness, and demonstrates effectiveness with limited training data.

Findings

Achieves high accuracy on SNLI while providing span-level explanations.

Improves out-of-distribution performance with fewer training examples.

Verifies model decisions align with human explanations.

Abstract

Current Natural Language Inference (NLI) models achieve impressive results, sometimes outperforming humans when evaluating on in-distribution test sets. However, as these models are known to learn from annotation artefacts and dataset biases, it is unclear to what extent the models are learning the task of NLI instead of learning from shallow heuristics in their training data. We address this issue by introducing a logical reasoning framework for NLI, creating highly transparent model decisions that are based on logical rules. Unlike prior work, we show that improved interpretability can be achieved without decreasing the predictive accuracy. We almost fully retain performance on SNLI, while also identifying the exact hypothesis spans that are responsible for each model prediction. Using the e-SNLI human explanations, we verify that our model makes sensible decisions at a span level, despite not using any span labels during training. We can further improve model performance and span-level decisions by using the e-SNLI explanations during training. Finally, our model is more robust in a reduced data setting. When training with only 1,000 examples, out-of-distribution performance improves on the MNLI matched and mismatched validation sets by 13% and 16% relative to the baseline. Training with fewer observations yields further improvements, both in-distribution and out-of-distribution.