Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning

TL;DR

This paper introduces invertible decoding functions in unsupervised sentence representation learning, enabling the decoder to be reused as an encoder, which improves the quality and transferability of sentence embeddings.

Contribution

It proposes a novel approach to utilize the decoder post-training by designing invertible decoding functions, enhancing sentence representation quality.

Findings

Inverse decoding functions serve as effective encoders.

Ensemble of encoder and inverse decoder improves generalization.

The method achieves better transferability of sentence representations.

Abstract

The encoder-decoder models for unsupervised sentence representation learning tend to discard the decoder after being trained on a large unlabelled corpus, since only the encoder is needed to map the input sentence into a vector representation. However, parameters learnt in the decoder also contain useful information about language. In order to utilise the decoder after learning, we present two types of decoding functions whose inverse can be easily derived without expensive inverse calculation. Therefore, the inverse of the decoding function serves as another encoder that produces sentence representations. We show that, with careful design of the decoding functions, the model learns good sentence representations, and the ensemble of the representations produced from the encoder and the inverse of the decoder demonstrate even better generalisation ability and solid transferability.