Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding

TL;DR

This paper introduces ConPVP, a novel contrastive learning approach for unsupervised sentence embeddings that leverages prompt-derived semantic prototypes to improve semantic representation quality.

Contribution

It proposes a new method using prompt-derived prototypes and a prototypical contrastive loss for better unsupervised sentence embedding, outperforming existing methods.

Findings

Improved performance on semantic textual similarity tasks.

Effective in transfer and clustering tasks.

Outperforms strong baseline models.

Abstract

Contrastive learning has become a new paradigm for unsupervised sentence embeddings. Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In this paper, we propose a novel Contrastive learning method with Prompt-derived Virtual semantic Prototypes (ConPVP). Specifically, with the help of prompts, we construct virtual semantic prototypes to each instance, and derive negative prototypes by using the negative form of the prompts. Using a prototypical contrastive loss, we enforce the anchor sentence embedding to be close to its corresponding semantic prototypes, and far apart from the negative prototypes as well as the prototypes of other sentences. Extensive experimental results on semantic textual similarity, transfer, and clustering tasks demonstrate the effectiveness of our proposed model compared to strong baselines. Code is available at https://github.com/lemon0830/promptCSE.