Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity

TL;DR

This paper introduces Pcc-tuning, a novel method using Pearson's correlation as a loss function, to surpass the existing performance ceiling of contrastive learning in semantic textual similarity tasks.

Contribution

Pcc-tuning is the first approach to effectively break the contrastive learning performance ceiling by optimizing Pearson's correlation in sentence embeddings.

Findings

Pcc-tuning exceeds previous state-of-the-art scores in STS benchmarks.

It requires only a small amount of annotated data.

The method achieves significant improvements over contrastive learning approaches.

Abstract

Semantic Textual Similarity (STS) constitutes a critical research direction in computational linguistics and serves as a key indicator of the encoding capabilities of embedding models. Driven by advances in pre-trained language models and contrastive learning, leading sentence representation methods have reached an average Spearman's correlation score of approximately 86 across seven STS benchmarks in SentEval. However, further progress has become increasingly marginal, with no existing method attaining an average score higher than 86.5 on these tasks. This paper conducts an in-depth analysis of this phenomenon and concludes that the upper limit for Spearman's correlation scores under contrastive learning is 87.5. To transcend this ceiling, we propose an innovative approach termed Pcc-tuning, which employs Pearson's correlation coefficient as a loss function to refine model performance beyond contrastive learning. Experimental results demonstrate that Pcc-tuning can markedly surpass previous state-of-the-art strategies with only a minimal amount of fine-grained annotated samples.