Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework

TL;DR

This paper introduces GenSE, a semi-supervised framework for sentence embedding that synthesizes and filters sentence pairs from unlabeled data, improving performance on semantic similarity and domain adaptation tasks.

Contribution

The paper presents a novel semi-supervised approach combining generation, discrimination, and contrastive learning for sentence embeddings, outperforming existing methods.

Findings

Achieves 85.19 average correlation on STS datasets

Significantly improves domain adaptation performance

Outperforms state-of-the-art sentence embedding methods

Abstract

Most sentence embedding techniques heavily rely on expensive human-annotated sentence pairs as the supervised signals. Despite the use of large-scale unlabeled data, the performance of unsupervised methods typically lags far behind that of the supervised counterparts in most downstream tasks. In this work, we propose a semi-supervised sentence embedding framework, GenSE, that effectively leverages large-scale unlabeled data. Our method include three parts: 1) Generate: A generator/discriminator model is jointly trained to synthesize sentence pairs from open-domain unlabeled corpus; 2) Discriminate: Noisy sentence pairs are filtered out by the discriminator to acquire high-quality positive and negative sentence pairs; 3) Contrast: A prompt-based contrastive approach is presented for sentence representation learning with both annotated and synthesized data. Comprehensive experiments show that GenSE achieves an average correlation score of 85.19 on the STS datasets and consistent performance improvement on four domain adaptation tasks, significantly surpassing the state-of-the-art methods and convincingly corroborating its effectiveness and generalization ability.Code, Synthetic data and Models available at https://github.com/MatthewCYM/GenSE.