Cropping outperforms dropout as an augmentation strategy for self-supervised training of text embeddings

TL;DR

This paper demonstrates that cropping augmentation significantly outperforms dropout in self-supervised fine-tuning of text embeddings, leading to higher quality representations especially for in-domain data.

Contribution

The study systematically compares cropping and dropout augmentation strategies, revealing cropping's superior effectiveness in self-supervised text embedding fine-tuning.

Findings

Cropping augmentation outperforms dropout in embedding quality.

Self-supervised fine-tuning yields high-quality in-domain embeddings after short training.

Embedding quality improves in the last transformer layers during fine-tuning.

Abstract

Text embeddings, i.e. vector representations of entire texts, play an important role in many NLP applications, such as retrieval-augmented generation, clustering, or visualizing collections of texts for data exploration. Currently, top-performing embedding models are derived from pre-trained language models via supervised contrastive fine-tuning. This fine-tuning strategy relies on an external notion of similarity and annotated data for generation of positive pairs. Here we study self-supervised fine-tuning and systematically compare the two most well-known augmentation strategies used for fine-tuning text embeddings models. We assess embedding quality on MTEB and additional in-domain evaluations and show that cropping augmentation strongly outperforms the dropout-based approach. We find that on out-of-domain data, the quality of resulting embeddings is substantially below the supervised state-of-the-art models, but for in-domain data, self-supervised fine-tuning can produce high-quality text embeddings after very short fine-tuning. Finally, we show that representation quality increases towards the last transformer layers, which undergo the largest change during fine-tuning; and that fine-tuning only those last layers is sufficient to reach similar embedding quality.