TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

TL;DR

TSDAE introduces a transformer-based unsupervised method for sentence embedding that outperforms previous models and generalizes well across multiple domains, reducing the need for labeled data.

Contribution

The paper presents TSDAE, a novel unsupervised transformer-based auto-encoder that achieves state-of-the-art results and demonstrates strong domain adaptation capabilities.

Findings

Outperforms previous approaches by up to 6.4 points

Achieves up to 93.1% of supervised performance

Excels in diverse domain evaluations

Abstract

Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming other approaches like Masked Language Model. A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.