Unsupervised Text Generation by Learning from Search

TL;DR

This paper introduces TGLS, an unsupervised text generation framework that combines search algorithms with generative models to improve quality, achieving competitive results with supervised methods.

Contribution

The paper proposes a novel unsupervised learning framework that iteratively combines search and generative modeling for natural language tasks.

Findings

TGLS outperforms baseline methods in paraphrase generation and text formalization.

Achieves comparable performance to supervised methods in paraphrase generation.

Demonstrates effectiveness on real-world NLP tasks.

Abstract

In this work, we present TGLS, a novel framework to unsupervised Text Generation by Learning from Search. We start by applying a strong search algorithm (in particular, simulated annealing) towards a heuristically defined objective that (roughly) estimates the quality of sentences. Then, a conditional generative model learns from the search results, and meanwhile smooth out the noise of search. The alternation between search and learning can be repeated for performance bootstrapping. We demonstrate the effectiveness of TGLS on two real-world natural language generation tasks, paraphrase generation and text formalization. Our model significantly outperforms unsupervised baseline methods in both tasks. Especially, it achieves comparable performance with the state-of-the-art supervised methods in paraphrase generation.