PAUSE: Positive and Annealed Unlabeled Sentence Embedding

TL;DR

PAUSE is a novel method for learning high-quality sentence embeddings from partially labeled data, outperforming some state-of-the-art techniques with less labeled data and reducing manual annotation effort.

Contribution

The paper introduces PAUSE, an end-to-end approach that effectively learns sentence embeddings from limited labeled data, applicable in industrial scenarios.

Findings

PAUSE achieves or surpasses state-of-the-art results on benchmark tasks.

PAUSE performs well with only a small fraction of labeled data.

PAUSE reduces the need for extensive manual annotation in real-world applications.

Abstract

Sentence embedding refers to a set of effective and versatile techniques for converting raw text into numerical vector representations that can be used in a wide range of natural language processing (NLP) applications. The majority of these techniques are either supervised or unsupervised. Compared to the unsupervised methods, the supervised ones make less assumptions about optimization objectives and usually achieve better results. However, the training requires a large amount of labeled sentence pairs, which is not available in many industrial scenarios. To that end, we propose a generic and end-to-end approach -- PAUSE (Positive and Annealed Unlabeled Sentence Embedding), capable of learning high-quality sentence embeddings from a partially labeled dataset. We experimentally show that PAUSE achieves, and sometimes surpasses, state-of-the-art results using only a small fraction of labeled sentence pairs on various benchmark tasks. When applied to a real industrial use case where labeled samples are scarce, PAUSE encourages us to extend our dataset without the liability of extensive manual annotation work.