SCD: Self-Contrastive Decorrelation for Sentence Embeddings

TL;DR

This paper introduces Self-Contrastive Decorrelation (SCD), a simple yet effective self-supervised method for sentence embeddings that achieves competitive results without relying on contrastive pairs, enhancing robustness and efficiency.

Contribution

The paper presents a novel self-supervised approach combining contrastive and decorrelation objectives, avoiding the need for contrastive pairs in training sentence embeddings.

Findings

Achieves comparable results to state-of-the-art methods on multiple benchmarks.

Does not require contrastive pairs for training, simplifying the process.

Demonstrates improved robustness over existing contrastive methods.

Abstract

In this paper, we propose Self-Contrastive Decorrelation (SCD), a self-supervised approach. Given an input sentence, it optimizes a joint self-contrastive and decorrelation objective. Learning a representation is facilitated by leveraging the contrast arising from the instantiation of standard dropout at different rates. The proposed method is conceptually simple yet empirically powerful. It achieves comparable results with state-of-the-art methods on multiple benchmarks without using contrastive pairs. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods.