NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals

TL;DR

This paper explores atomic hypothesis decomposition in natural language inference to analyze model reasoning, uncovering challenges in logical consistency and proposing a metric for inferential consistency.

Contribution

It introduces atomic decomposition for NLI and defeasible NLI, providing new insights into model reasoning and a method to measure inferential consistency.

Findings

LLMs struggle with logical consistency on atomic NLI tasks.

Atomic decomposition reveals model understanding limitations.

Proposes a metric for assessing inferential consistency.

Abstract

Decomposition of text into atomic propositions is a flexible framework allowing for the closer inspection of input and output text. We use atomic decomposition of hypotheses in two natural language reasoning tasks, traditional NLI and defeasible NLI, to form atomic sub-problems, or granular inferences that models must weigh when solving the overall problem. These atomic sub-problems serve as a tool to further understand the structure of both NLI and defeasible reasoning, probe a model's consistency and understanding of different inferences, and measure the diversity of examples in benchmark datasets. Our results indicate that LLMs still struggle with logical consistency on atomic NLI and defeasible NLI sub-problems. Lastly, we identify critical atomic sub-problems of defeasible NLI examples, or those that most contribute to the overall label, and propose a method to measure the inferential consistency of a model, a metric designed to capture the degree to which a model makes consistently correct or incorrect predictions about the same fact under different contexts.