Unsupervised Learning of Sentence Representations Using Sequence Consistency

TL;DR

This paper introduces ConsSent, an unsupervised method for learning sentence representations by enforcing sequence consistency constraints, leading to improved performance on various NLP tasks.

Contribution

It proposes a novel unsupervised approach that uses sequence consistency constraints and perturbation-based training to learn effective sentence encoders.

Findings

ConsSent outperforms strong unsupervised and supervised baselines.

Multitask training and ensemble methods further improve results.

The approach is effective across multiple transfer and linguistic tasks.

Abstract

Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens. We consider two classes of such constraints -- sequences that form a sentence and between two sequences that form a sentence when merged. We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, the latter being generated by randomly perturbing consistent examples in six different ways. Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.