Uncertain Natural Language Inference

TL;DR

This paper proposes Uncertain Natural Language Inference (UNLI), a probabilistic approach to NLI that predicts subjective likelihoods instead of categorical labels, enabling more nuanced understanding of language inference.

Contribution

It introduces UNLI, a new probabilistic NLI framework, and demonstrates how existing datasets can be adapted for scalar regression modeling of inference uncertainty.

Findings

Models approach human performance in predicting likelihoods.

Existing NLI data can be repurposed for probabilistic inference.

UNLI captures subtle differences in inference judgments.

Abstract

We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments. We demonstrate the feasibility of collecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise. We describe a direct scalar regression modeling approach, and find that existing categorically labeled NLI data can be used in pre-training. Our best models approach human performance, demonstrating models may be capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks.