SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives

TL;DR

This paper enhances contrastive learning for sentence embeddings by addressing dropout noise and feature corruption, leading to significant performance improvements on standard benchmarks.

Contribution

It introduces two novel methods to handle dropout noise and feature corruption, applicable to any contrastive learning models for sentence embeddings.

Findings

Achieved 1.8 point gain over SimCSE baseline.

Achieved 1.4 point gain over DiffCSE baseline.

Methods are generic and improve model robustness.

Abstract

This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.