Multiple Structural Priors Guided Self Attention Network for Language Understanding

TL;DR

This paper introduces MS-SAN, a self-attention network that incorporates multiple structural priors like word order and syntax trees through a multi-mask attention mechanism, improving language understanding performance.

Contribution

It proposes a novel multi-mask multi-head attention mechanism to embed diverse structural priors into SANs, enhancing their ability to model complex text structures.

Findings

MS-SAN outperforms strong baselines on two NLP tasks.

Incorporating multiple structural priors improves model accuracy.

The approach effectively captures hierarchical and syntactic information.

Abstract

Self attention networks (SANs) have been widely utilized in recent NLP studies. Unlike CNNs or RNNs, standard SANs are usually position-independent, and thus are incapable of capturing the structural priors between sequences of words. Existing studies commonly apply one single mask strategy on SANs for incorporating structural priors while failing at modeling more abundant structural information of texts. In this paper, we aim at introducing multiple types of structural priors into SAN models, proposing the Multiple Structural Priors Guided Self Attention Network (MS-SAN) that transforms different structural priors into different attention heads by using a novel multi-mask based multi-head attention mechanism. In particular, we integrate two categories of structural priors, including the sequential order and the relative position of words. For the purpose of capturing the latent hierarchical structure of the texts, we extract these information not only from the word contexts but also from the dependency syntax trees. Experimental results on two tasks show that MS-SAN achieves significant improvements against other strong baselines.