Backward and Forward Language Modeling for Constrained Sentence Generation

TL;DR

This paper introduces a backward and forward language model that allows for constrained sentence generation by ensuring a specific word appears in the generated text, regardless of its position.

Contribution

It proposes a novel RNN-based model that generates context around a given word, enabling constrained sentence generation at any position.

Findings

Generated texts are comparable in quality to traditional sequential language models.

The model effectively places a specified word at any position in the sentence.

Experimental results validate the approach's feasibility and quality.

Abstract

Recent language models, especially those based on recurrent neural networks (RNNs), make it possible to generate natural language from a learned probability. Language generation has wide applications including machine translation, summarization, question answering, conversation systems, etc. Existing methods typically learn a joint probability of words conditioned on additional information, which is (either statically or dynamically) fed to RNN's hidden layer. In many applications, we are likely to impose hard constraints on the generated texts, i.e., a particular word must appear in the sentence. Unfortunately, existing approaches could not solve this problem. In this paper, we propose a novel backward and forward language model. Provided a specific word, we use RNNs to generate previous words and future words, either simultaneously or asynchronously, resulting in two model variants. In this way, the given word could appear at any position in the sentence. Experimental results show that the generated texts are comparable to sequential LMs in quality.