Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE

TL;DR

This paper enhances contrastive learning for sentence embeddings by integrating hard negative mining with a focal-InfoNCE loss, leading to improved performance on semantic similarity benchmarks.

Contribution

It introduces a novel unsupervised framework combining SimCSE with focal-InfoNCE to better utilize hard negatives in contrastive learning.

Findings

Improved Spearman's correlation on STS benchmarks

Enhanced representation alignment and uniformity

Outperforms baseline methods in sentence similarity tasks

Abstract

The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives. Experimentation on various STS benchmarks shows that our method improves sentence embeddings in terms of Spearman's correlation and representation alignment and uniformity.