EIT: Enhanced Interactive Transformer

TL;DR

This paper introduces EMHA, an improved multi-head self-attention mechanism that enhances consensus among heads and relaxes the complementarity constraints, leading to better performance on various language tasks.

Contribution

The paper proposes EMHA, which incorporates inner- and cross-subspace interactions to improve multi-head self-attention by balancing complementarity and consensus.

Findings

Outperforms existing models on multiple language tasks

Achieves superior results with minimal increase in model size

Demonstrates the effectiveness of enhanced consensus mechanisms

Abstract

Two principles: the complementary principle and the consensus principle are widely acknowledged in the literature of multi-view learning. However, the current design of multi-head self-attention, an instance of multi-view learning, prioritizes the complementarity while ignoring the consensus. To address this problem, we propose an enhanced multi-head self-attention (EMHA). First, to satisfy the complementary principle, EMHA removes the one-to-one mapping constraint among queries and keys in multiple subspaces and allows each query to attend to multiple keys. On top of that, we develop a method to fully encourage consensus among heads by introducing two interaction models, namely inner-subspace interaction and cross-subspace interaction. Extensive experiments on a wide range of language tasks (e.g., machine translation, abstractive summarization and grammar correction, language modeling), show its superiority, with a very modest increase in model size. Our code would be available at: https://github.com/zhengkid/EIT-Enhanced-Interactive-Transformer.