Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

TL;DR

This paper introduces IS-CSE, a novel contrastive learning method that smooths sentence embeddings by aggregating similar instances, leading to improved unsupervised sentence embedding performance.

Contribution

The study proposes a new instance smoothing technique for contrastive learning that enhances generalization in unsupervised sentence embeddings.

Findings

Achieves state-of-the-art results on STS tasks with BERT and RoBERTa models.

Improves Spearman's correlation by over 2% on average.

Demonstrates better generalization through embedding smoothing.

Abstract

Contrastive learning-based methods, such as unsup-SimCSE, have achieved state-of-the-art (SOTA) performances in learning unsupervised sentence embeddings. However, in previous studies, each embedding used for contrastive learning only derived from one sentence instance, and we call these embeddings instance-level embeddings. In other words, each embedding is regarded as a unique class of its own, whichmay hurt the generalization performance. In this study, we propose IS-CSE (instance smoothing contrastive sentence embedding) to smooth the boundaries of embeddings in the feature space. Specifically, we retrieve embeddings from a dynamic memory buffer according to the semantic similarity to get a positive embedding group. Then embeddings in the group are aggregated by a self-attention operation to produce a smoothed instance embedding for further analysis. We evaluate our method on standard semantic text similarity (STS) tasks and achieve an average of 78.30%, 79.47%, 77.73%, and 79.42% Spearman's correlation on the base of BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large respectively, a 2.05%, 1.06%, 1.16% and 0.52% improvement compared to unsup-SimCSE.