Sentence Representation Learning with Generative Objective rather than Contrastive Objective

TL;DR

This paper introduces a generative self-supervised learning approach for sentence representation that models intra-sentence structure through phrase reconstruction, outperforming contrastive methods on semantic tasks.

Contribution

It proposes a novel phrase-based generative objective for sentence embedding, addressing interpretability and performance issues of contrastive learning methods.

Findings

Outperforms contrastive methods on STS benchmarks

Improves downstream semantic retrieval and reranking tasks

Achieves significant performance gains in sentence representation learning

Abstract

Though offering amazing contextualized token-level representations, current pre-trained language models take less attention on accurately acquiring sentence-level representation during their self-supervised pre-training. However, contrastive objectives which dominate the current sentence representation learning bring little linguistic interpretability and no performance guarantee on downstream semantic tasks. We instead propose a novel generative self-supervised learning objective based on phrase reconstruction. To overcome the drawbacks of previous generative methods, we carefully model intra-sentence structure by breaking down one sentence into pieces of important phrases. Empirical studies show that our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods not only on the STS benchmarks, but also on downstream semantic retrieval and reranking tasks. Our code is available at https://github.com/chengzhipanpan/PaSeR.