Encodings of Source Syntax: Similarities in NMT Representations Across Target Languages

TL;DR

This study shows that NMT encoder representations capture similar source syntax across different target languages and outperform RNNs on syntactic tasks, though they differ from traditional PCFG parsers in the syntactic information learned.

Contribution

It demonstrates that NMT encoders learn target-language-independent source syntax and outperform RNNs trained directly on syntactic prediction tasks.

Findings

NMT encoders learn similar source syntax across target languages.

NMT encoders outperform RNNs on constituent label prediction.

PCFG parsers perform well on sentences where RNNs perform poorly.

Abstract

We train neural machine translation (NMT) models from English to six target languages, using NMT encoder representations to predict ancestor constituent labels of source language words. We find that NMT encoders learn similar source syntax regardless of NMT target language, relying on explicit morphosyntactic cues to extract syntactic features from source sentences. Furthermore, the NMT encoders outperform RNNs trained directly on several of the constituent label prediction tasks, suggesting that NMT encoder representations can be used effectively for natural language tasks involving syntax. However, both the NMT encoders and the directly-trained RNNs learn substantially different syntactic information from a probabilistic context-free grammar (PCFG) parser. Despite lower overall accuracy scores, the PCFG often performs well on sentences for which the RNN-based models perform poorly, suggesting that RNN architectures are constrained in the types of syntax they can learn.