PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings

TL;DR

This paper introduces Peer-Contrastive Learning (PCL), a novel method that leverages diverse augmentations and peer-positive contrast to improve unsupervised sentence embeddings, addressing biases from mono-augmentation strategies.

Contribution

PCL is the first to utilize group-level diverse positives and peer-network cooperation for unsupervised sentence embedding learning, enhancing robustness and quality.

Findings

PCL outperforms existing methods on STS benchmarks.

Diverse augmentations improve embedding quality.

Peer-contrastive learning reduces bias in sentence representations.

Abstract

Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semantically-close positive instance to construct contrastive pairs. Nonetheless, existing approaches usually depend on a mono-augmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings. A straightforward solution is resorting to more diverse positives from a multi-augmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field. As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL performs peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations. Experiments on STS benchmarks verify the effectiveness of PCL against its competitors in unsupervised sentence embeddings.