Debiased Contrastive Learning of Unsupervised Sentence Representations

TL;DR

This paper introduces DCLR, a debiased contrastive learning framework that improves unsupervised sentence representations by addressing negative sampling bias, leading to better semantic similarity performance.

Contribution

The paper proposes a novel instance weighting and noise-based negative sampling method to mitigate negative sampling bias in contrastive learning for sentence representations.

Findings

Outperforms baseline methods on seven semantic textual similarity tasks.

Effectively reduces false negatives and improves representation uniformity.

Demonstrates robustness across various unsupervised settings.

Abstract

Recently, contrastive learning has been shown to be effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignment while push apart irrelevant negatives for the uniformity of the whole representation space. However, previous works mostly adopt in-batch negatives or sample from training data at random. Such a way may cause the sampling bias that improper negatives (e.g. false negatives and anisotropy representations) are used to learn sentence representations, which will hurt the uniformity of the representation space. To address it, we present a new framework \textbf{DCLR} (\underline{D}ebiased \underline{C}ontrastive \underline{L}earning of unsupervised sentence \underline{R}epresentations) to alleviate the influence of these improper negatives. In DCLR, we design an instance weighting method to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. Experiments on seven semantic textual similarity tasks show that our approach is more effective than competitive baselines. Our code and data are publicly available at the link: \textcolor{blue}{\url{https://github.com/RUCAIBox/DCLR}}.