Towards Two-Dimensional Sequence to Sequence Model in Neural Machine Translation

TL;DR

This paper introduces a novel 2D sequence-to-sequence model for neural machine translation using multi-dimensional LSTM, which models source and target sentences as a 2D grid, improving translation quality over traditional 1D models.

Contribution

It proposes a 2D NMT architecture employing MDLSTM to better capture source-target alignment, extending beyond traditional 1D sequence models.

Findings

Consistent improvements on WMT 2017 German-English translation tasks.

Demonstrates the effectiveness of 2D modeling over attention-based models.

Abstract

This work investigates an alternative model for neural machine translation (NMT) and proposes a novel architecture, where we employ a multi-dimensional long short-term memory (MDLSTM) for translation modeling. In the state-of-the-art methods, source and target sentences are treated as one-dimensional sequences over time, while we view translation as a two-dimensional (2D) mapping using an MDLSTM layer to define the correspondence between source and target words. We extend beyond the current sequence to sequence backbone NMT models to a 2D structure in which the source and target sentences are aligned with each other in a 2D grid. Our proposed topology shows consistent improvements over attention-based sequence to sequence model on two WMT 2017 tasks, German$\leftrightarrow$English.