Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network

TL;DR

Deps-SAN introduces a dependency-scaled self-attention mechanism in Transformer-based NMT to incorporate syntactic knowledge explicitly, improving translation quality without heavy additional workloads.

Contribution

The paper proposes a parameter-free, dependency-scaled self-attention network that effectively integrates syntactic constraints into Transformer-based NMT models.

Findings

Improved translation performance on IWSLT14 and WMT16 benchmarks.

Effective incorporation of syntactic constraints enhances model accuracy.

Knowledge sparsing techniques prevent overfitting to dependency noise.

Abstract

Syntax knowledge contributes its powerful strength in Neural machine translation (NMT) tasks. Early NMT works supposed that syntax details can be automatically learned from numerous texts via attention networks. However, succeeding researches pointed out that limited by the uncontrolled nature of attention computation, the NMT model requires an external syntax to capture the deep syntactic awareness. Although existing syntax-aware NMT methods have born great fruits in combining syntax, the additional workloads they introduced render the model heavy and slow. Particularly, these efforts scarcely involve the Transformer-based NMT and modify its core self-attention network (SAN). To this end, we propose a parameter-free, Dependency-scaled Self-Attention Network (Deps-SAN) for syntax-aware Transformer-based NMT. A quantified matrix of dependency closeness between tokens is constructed to impose explicit syntactic constraints into the SAN for learning syntactic details and dispelling the dispersion of attention distributions. Two knowledge sparsing techniques are further integrated to avoid the model overfitting the dependency noises introduced by the external parser. Experiments and analyses on IWSLT14 German-to-English and WMT16 German-to-English benchmark NMT tasks verify the effectiveness of our approach.