Collective Human Opinions in Semantic Textual Similarity

TL;DR

This paper introduces USTS, a new Chinese dataset for semantic textual similarity that captures human disagreement and semantic vagueness, highlighting limitations of current models in representing opinion variance.

Contribution

The paper presents USTS, the first uncertainty-aware STS dataset with detailed annotations, and analyzes the inadequacy of existing models to capture human opinion variance.

Findings

Existing benchmarks mask opinion disagreement by averaging ratings.

Current models do not effectively capture individual opinion variance.

USTS dataset reveals the complexity of human semantic judgments.

Abstract

Despite the subjective nature of semantic textual similarity (STS) and pervasive disagreements in STS annotation, existing benchmarks have used averaged human ratings as the gold standard. Averaging masks the true distribution of human opinions on examples of low agreement, and prevents models from capturing the semantic vagueness that the individual ratings represent. In this work, we introduce USTS, the first Uncertainty-aware STS dataset with ~15,000 Chinese sentence pairs and 150,000 labels, to study collective human opinions in STS. Analysis reveals that neither a scalar nor a single Gaussian fits a set of observed judgements adequately. We further show that current STS models cannot capture the variance caused by human disagreement on individual instances, but rather reflect the predictive confidence over the aggregate dataset.