Multi-view Sentence Representation Learning

TL;DR

This paper introduces a multi-view sentence representation learning framework combining RNN and linear models, leveraging adjacent sentence context for self-supervision, resulting in improved and transferable sentence embeddings.

Contribution

It proposes a novel multi-view learning approach using asymmetric models and context agreement, enhancing sentence representations over single-view methods.

Findings

Multi-view training improves sentence embeddings.

Combining views yields further performance gains.

Embeddings transfer well to downstream tasks.

Abstract

Multi-view learning can provide self-supervision when different views are available of the same data. The distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we create a unified multi-view sentence representation learning framework, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model, and the training objective is to maximise the agreement specified by the adjacent context information between two views. We show that, after training, the vectors produced from our multi-view training provide improved representations over the single-view training, and the combination of different views gives further representational improvement and demonstrates solid transferability on standard downstream tasks.