Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

TL;DR

This paper empirically evaluates neural machine translation architectures, revealing that self-attention models excel in semantic feature extraction but do not outperform RNNs in modeling long-range dependencies like subject-verb agreement.

Contribution

It provides a targeted empirical comparison of RNNs, CNNs, and self-attention networks, challenging assumptions about their capabilities in long-range dependency modeling.

Findings

Self-attention networks do not outperform RNNs in long-distance subject-verb agreement.

Self-attention networks outperform RNNs and CNNs in word sense disambiguation.

CNNs and self-attention models excel at semantic feature extraction.

Abstract

Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation.