Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss

TL;DR

This paper introduces a novel regression framework with Translated ReLU and Smooth K2 Loss for Semantic Textual Similarity, addressing limitations of contrastive learning and classification-based methods, and demonstrating strong results across multiple benchmarks.

Contribution

It proposes a new regression-based approach with innovative loss functions to better model nuanced semantic similarities in STS tasks.

Findings

Achieves competitive performance on seven STS benchmarks.

Effectively models fine-grained semantic similarity levels.

Potential to enhance contrastive learning pre-trained models.

Abstract

Since the introduction of BERT and RoBERTa, research on Semantic Textual Similarity (STS) has made groundbreaking progress. Particularly, the adoption of contrastive learning has substantially elevated state-of-the-art performance across various STS benchmarks. However, contrastive learning categorizes text pairs as either semantically similar or dissimilar, failing to leverage fine-grained annotated information and necessitating large batch sizes to prevent model collapse. These constraints pose challenges for researchers engaged in STS tasks that involve nuanced similarity levels or those with limited computational resources, compelling them to explore alternatives like Sentence-BERT. Despite its efficiency, Sentence-BERT tackles STS tasks from a classification perspective, overlooking the progressive nature of semantic relationships, which results in suboptimal performance. To bridge this gap, this paper presents an innovative regression framework and proposes two simple yet effective loss functions: Translated ReLU and Smooth K2 Loss. Experimental results demonstrate that our method achieves convincing performance across seven established STS benchmarks and offers the potential for further optimization of contrastive learning pre-trained models.