Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference

TL;DR

This paper investigates the logical properties of Natural Language Inference (NLI) by analyzing label interpretations and meta-inferential consistency, revealing insights into what the SNLI dataset encodes about inference.

Contribution

It formulates three interpretations of NLI labels and analyzes their implications using SNLI data and LLM-generated items to understand the logical properties captured by models.

Findings

Different label readings imply distinct logical properties.

Models show varying consistency depending on the label interpretation.

SNLI encodes specific meta-inferential relations aligned with one interpretation.

Abstract

Natural Language Inference (NLI) has been an important task for evaluating language models for Natural Language Understanding, but the logical properties of the task are poorly understood and often mischaracterized. Understanding the notion of inference captured by NLI is key to interpreting model performance on the task. In this paper we formulate three possible readings of the NLI label set and perform a comprehensive analysis of the meta-inferential properties they entail. Focusing on the SNLI dataset, we exploit (1) NLI items with shared premises and (2) items generated by LLMs to evaluate models trained on SNLI for meta-inferential consistency and derive insights into which reading of the logical relations is encoded by the dataset.