RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank

TL;DR

RankCSE introduces an unsupervised learning method that enhances sentence representations by integrating ranking consistency and distillation, effectively capturing fine-grained semantic relevance for improved NLP task performance.

Contribution

The paper proposes a novel unsupervised sentence embedding approach that incorporates ranking information through consistency and distillation, addressing limitations of contrastive learning.

Findings

Outperforms state-of-the-art baselines on semantic similarity tasks.

Effectively captures fine-grained ranking among sentences.

Improves transfer learning performance across NLP tasks.

Abstract

Unsupervised sentence representation learning is one of the fundamental problems in natural language processing with various downstream applications. Recently, contrastive learning has been widely adopted which derives high-quality sentence representations by pulling similar semantics closer and pushing dissimilar ones away. However, these methods fail to capture the fine-grained ranking information among the sentences, where each sentence is only treated as either positive or negative. In many real-world scenarios, one needs to distinguish and rank the sentences based on their similarities to a query sentence, e.g., very relevant, moderate relevant, less relevant, irrelevant, etc. In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. In particular, we learn semantically discriminative sentence representations by simultaneously ensuring ranking consistency between two representations with different dropout masks, and distilling listwise ranking knowledge from the teacher. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks. Experimental results demonstrate the superior performance of our approach over several state-of-the-art baselines.