Word Ordering Without Syntax

TL;DR

This paper demonstrates that simple n-gram and LSTM language models can effectively recover sentence word order without relying on syntactic structure, outperforming syntactic models significantly.

Contribution

It shows that non-syntactic language models, especially LSTMs, can surpass syntactic models in word order recovery tasks.

Findings

LSTM models outperform syntactic models by 11.5 BLEU points

Heuristic n-gram models achieve strong results

Additional data and larger beams improve performance

Abstract

Recent work on word ordering has argued that syntactic structure is important, or even required, for effectively recovering the order of a sentence. We find that, in fact, an n-gram language model with a simple heuristic gives strong results on this task. Furthermore, we show that a long short-term memory (LSTM) language model is even more effective at recovering order, with our basic model outperforming a state-of-the-art syntactic model by 11.5 BLEU points. Additional data and larger beams yield further gains, at the expense of training and search time.