An Intrinsic Nearest Neighbor Analysis of Neural Machine Translation Architectures

TL;DR

This paper investigates the hidden states of transformer and recurrent neural machine translation models using a nearest neighbors approach to analyze their ability to capture lexical semantics and syntactic structures.

Contribution

It introduces an intrinsic analysis method comparing transformer and recurrent models based on nearest neighbor relationships, revealing differences in semantic and syntactic encoding.

Findings

Transformers better capture lexical semantics.

Recurrent models' backward layer encodes more semantics.

Recurrent models' forward layer encodes more context.

Abstract

Earlier approaches indirectly studied the information captured by the hidden states of recurrent and non-recurrent neural machine translation models by feeding them into different classifiers. In this paper, we look at the encoder hidden states of both transformer and recurrent machine translation models from the nearest neighbors perspective. We investigate to what extent the nearest neighbors share information with the underlying word embeddings as well as related WordNet entries. Additionally, we study the underlying syntactic structure of the nearest neighbors to shed light on the role of syntactic similarities in bringing the neighbors together. We compare transformer and recurrent models in a more intrinsic way in terms of capturing lexical semantics and syntactic structures, in contrast to extrinsic approaches used by previous works. In agreement with the extrinsic evaluations in the earlier works, our experimental results show that transformers are superior in capturing lexical semantics, but not necessarily better in capturing the underlying syntax. Additionally, we show that the backward recurrent layer in a recurrent model learns more about the semantics of words, whereas the forward recurrent layer encodes more context.