Embracing Ambiguity: Shifting the Training Target of NLI Models

TL;DR

This paper investigates training NLI models on label distributions reflecting annotator ambiguity, demonstrating improved performance and representation learning by embracing inherent dataset ambiguity.

Contribution

It introduces a novel training approach using ambiguity distributions, along with the AmbiNLI dataset, to better capture linguistic ambiguity in NLI models.

Findings

Reduced ChaosNLI divergence scores after fine-tuning on AmbiNLI

Models trained on ambiguity distributions outperform those trained on gold labels

Enhanced downstream task performance and representation quality

Abstract

Natural Language Inference (NLI) datasets contain examples with highly ambiguous labels. While many research works do not pay much attention to this fact, several recent efforts have been made to acknowledge and embrace the existence of ambiguity, such as UNLI and ChaosNLI. In this paper, we explore the option of training directly on the estimated label distribution of the annotators in the NLI task, using a learning loss based on this ambiguity distribution instead of the gold-labels. We prepare AmbiNLI, a trial dataset obtained from readily available sources, and show it is possible to reduce ChaosNLI divergence scores when finetuning on this data, a promising first step towards learning how to capture linguistic ambiguity. Additionally, we show that training on the same amount of data but targeting the ambiguity distribution instead of gold-labels can result in models that achieve higher performance and learn better representations for downstream tasks.