C-STS: Conditional Semantic Textual Similarity

TL;DR

This paper introduces Conditional STS (C-STS), a new task that measures sentence similarity based on specified natural language conditions, reducing ambiguity and enabling detailed model evaluation.

Contribution

It proposes the C-STS task, creates a large dataset, and evaluates state-of-the-art models, highlighting challenges and advancing semantic similarity assessment.

Findings

State-of-the-art models perform poorly on C-STS with correlations below 50

C-STS reduces subjectivity in semantic similarity measurement

Provides a large dataset and code for future research

Abstract

Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information retrieval and natural language understanding. However, sentence similarity can be inherently ambiguous, depending on the specific aspect of interest. We resolve this ambiguity by proposing a novel task called Conditional STS (C-STS) which measures sentences' similarity conditioned on an feature described in natural language (hereon, condition). As an example, the similarity between the sentences "The NBA player shoots a three-pointer." and "A man throws a tennis ball into the air to serve." is higher for the condition "The motion of the ball" (both upward) and lower for "The size of the ball" (one large and one small). C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity of STS and (2) enables fine-grained language model evaluation through diverse natural language conditions. We put several state-of-the-art models to the test, and even those performing well on STS (e.g. SimCSE, Flan-T5, and GPT-4) find C-STS challenging; all with Spearman correlation scores below 50. To encourage a more comprehensive evaluation of semantic similarity and natural language understanding, we make nearly 19K C-STS examples and code available for others to train and test their models.