Language as a Latent Sequence: deep latent variable models for semi-supervised paraphrase generation

TL;DR

This paper introduces a semi-supervised paraphrase generation framework using deep latent variable models, combining unsupervised and supervised approaches with novel training schemes to improve performance with limited labeled data.

Contribution

The paper proposes a novel unsupervised latent sequence model (VSAR), a dual directional learning approach (DDL), and a two-stage training scheme (KRL) to enhance semi-supervised paraphrase generation.

Findings

Competitive performance against state-of-the-art supervised models on full data.

Significant improvements over supervised baselines with limited labeled data.

Effective semi-supervised learning demonstrated through empirical evaluations.

Abstract

This paper explores deep latent variable models for semi-supervised paraphrase generation, where the missing target pair for unlabelled data is modelled as a latent paraphrase sequence. We present a novel unsupervised model named variational sequence auto-encoding reconstruction (VSAR), which performs latent sequence inference given an observed text. To leverage information from text pairs, we additionally introduce a novel supervised model we call dual directional learning (DDL), which is designed to integrate with our proposed VSAR model. Combining VSAR with DDL (DDL+VSAR) enables us to conduct semi-supervised learning. Still, the combined model suffers from a cold-start problem. To further combat this issue, we propose an improved weight initialisation solution, leading to a novel two-stage training scheme we call knowledge-reinforced-learning (KRL). Our empirical evaluations suggest that the combined model yields competitive performance against the state-of-the-art supervised baselines on complete data. Furthermore, in scenarios where only a fraction of the labelled pairs are available, our combined model consistently outperforms the strong supervised model baseline (DDL) by a significant margin (p <.05; Wilcoxon test). Our code is publicly available at "https://github.com/jialin-yu/latent-sequence-paraphrase".