Modeling Source Syntax for Neural Machine Translation

TL;DR

This paper demonstrates that explicitly incorporating source syntax via linearized parse trees into neural machine translation models significantly improves translation quality, with the simplest mixed RNN encoder achieving the best results.

Contribution

It introduces three novel syntactic encoders for NMT that explicitly utilize source syntax, leading to notable translation improvements.

Findings

All three syntactic encoders improve translation accuracy.

The simplest Mixed RNN encoder performs best with a 1.4 BLEU point gain.

Source syntax benefits NMT as shown by detailed analysis.

Abstract

Even though a linguistics-free sequence to sequence model in neural machine translation (NMT) has certain capability of implicitly learning syntactic information of source sentences, this paper shows that source syntax can be explicitly incorporated into NMT effectively to provide further improvements. Specifically, we linearize parse trees of source sentences to obtain structural label sequences. On the basis, we propose three different sorts of encoders to incorporate source syntax into NMT: 1) Parallel RNN encoder that learns word and label annotation vectors parallelly; 2) Hierarchical RNN encoder that learns word and label annotation vectors in a two-level hierarchy; and 3) Mixed RNN encoder that stitchingly learns word and label annotation vectors over sequences where words and labels are mixed. Experimentation on Chinese-to-English translation demonstrates that all the three proposed syntactic encoders are able to improve translation accuracy. It is interesting to note that the simplest RNN encoder, i.e., Mixed RNN encoder yields the best performance with an significant improvement of 1.4 BLEU points. Moreover, an in-depth analysis from several perspectives is provided to reveal how source syntax benefits NMT.