CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass

TL;DR

CSE-SFP is a novel unsupervised sentence representation method for generative pre-trained language models that achieves high-quality embeddings with only a single forward pass, reducing training time and memory usage.

Contribution

It introduces CSE-SFP, a single-pass contrastive learning framework tailored for decoder-only generative PLMs, addressing efficiency and quality in unsupervised sentence embedding.

Findings

Produces higher-quality embeddings than existing methods

Reduces training time and memory consumption

Provides robust evaluation metrics for semantic properties

Abstract

As a fundamental task in Information Retrieval and Computational Linguistics, sentence representation has profound implications for a wide range of practical applications such as text clustering, content analysis, question-answering systems, and web search. Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field, particularly through unsupervised embedding derivation methods centered on discriminative PLMs like BERT. However, due to time and computational constraints, few efforts have attempted to integrate unsupervised sentence representation with generative PLMs, which typically possess much larger parameter sizes. Given that state-of-the-art models in both academia and industry are predominantly based on generative architectures, there is a pressing need for an efficient unsupervised text representation framework tailored to decoder-only PLMs. To address this concern, we propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models. Compared to existing strategies, CSE-SFP requires only a single forward pass to perform effective unsupervised contrastive learning. Rigorous experimentation demonstrates that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption. Furthermore, we introduce two ratio metrics that jointly assess alignment and uniformity, thereby providing a more robust means for evaluating the semantic spatial properties of encoding models.