Trimming and Improving Skip-thought Vectors

TL;DR

This paper introduces a streamlined, faster version of the skip-thought model that maintains its effectiveness in learning sentence representations by simplifying the architecture and optimizing training techniques.

Contribution

The paper proposes a trimmed skip-thought model with a single decoder, a connection layer for better generalization, and emphasizes the importance of word embedding initialization.

Findings

The trimmed model is faster and lighter than the original.

It performs equally well on various semantic and text classification tasks.

The model benefits from improved generalization and initialization strategies.

Abstract

The skip-thought model has been proven to be effective at learning sentence representations and capturing sentence semantics. In this paper, we propose a suite of techniques to trim and improve it. First, we validate a hypothesis that, given a current sentence, inferring the previous and inferring the next sentence provide similar supervision power, therefore only one decoder for predicting the next sentence is preserved in our trimmed skip-thought model. Second, we present a connection layer between encoder and decoder to help the model to generalize better on semantic relatedness tasks. Third, we found that a good word embedding initialization is also essential for learning better sentence representations. We train our model unsupervised on a large corpus with contiguous sentences, and then evaluate the trained model on 7 supervised tasks, which includes semantic relatedness, paraphrase detection, and text classification benchmarks. We empirically show that, our proposed model is a faster, lighter-weight and equally powerful alternative to the original skip-thought model.