Convolutional Self-Attention Networks

TL;DR

This paper introduces convolutional self-attention networks that enhance local dependencies and feature interactions in SANs, leading to improved machine translation performance without increasing model parameters.

Contribution

The paper proposes convolutional self-attention networks that strengthen local dependencies and feature interactions, outperforming Transformer baselines without adding extra parameters.

Findings

Outperforms Transformer baseline in machine translation tasks

Enhances locality modeling in self-attention networks

No additional parameters introduced

Abstract

Self-attention networks (SANs) have drawn increasing interest due to their high parallelization in computation and flexibility in modeling dependencies. SANs can be further enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks, which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the interaction between features extracted by multiple attention heads. Experimental results of machine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer baseline and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters.