Spiral Language Modeling

TL;DR

Spiral Language Modeling (SLM) introduces a flexible text generation approach that constructs sentences starting from arbitrary tokens, enhancing diversity, quality, and robustness in neural machine translation tasks.

Contribution

SLM enables non-linear text generation starting from any token, improving translation quality and robustness compared to traditional left-to-right methods.

Findings

Up to 4.7 BLEU score improvements in NMT tasks

Enhanced diversity and quality of generated text

Improved robustness in low-resource scenarios

Abstract

In almost all text generation applications, word sequences are constructed in a left-to-right (L2R) or right-to-left (R2L) manner, as natural language sentences are written either L2R or R2L. However, we find that the natural language written order is not essential for text generation. In this paper, we propose Spiral Language Modeling (SLM), a general approach that enables one to construct natural language sentences beyond the L2R and R2L order. SLM allows one to form natural language text by starting from an arbitrary token inside the result text and expanding the rest tokens around the selected ones. It makes the decoding order a new optimization objective besides the language model perplexity, which further improves the diversity and quality of the generated text. Furthermore, SLM makes it possible to manipulate the text construction process by selecting a proper starting token. SLM also introduces generation orderings as additional regularization to improve model robustness in low-resource scenarios. Experiments on 8 widely studied Neural Machine Translation (NMT) tasks show that SLM is constantly effective with up to 4.7 BLEU increase comparing to the conventional L2R decoding approach.