Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together

TL;DR

This paper introduces MTSA, a novel self-attention mechanism that efficiently models pairwise and global dependencies, outperforming previous models while maintaining low memory and computational costs.

Contribution

The paper proposes MTSA, a tensorized self-attention method that captures diverse dependencies with improved efficiency and expressiveness compared to existing models.

Findings

Achieves state-of-the-art or competitive results on nine NLP benchmarks.

Demonstrates significant memory and time efficiency over traditional models.

Effectively models both local and long-range dependencies in NLP tasks.

Abstract

Neural networks equipped with self-attention have parallelizable computation, light-weight structure, and the ability to capture both long-range and local dependencies. Further, their expressive power and performance can be boosted by using a vector to measure pairwise dependency, but this requires to expand the alignment matrix to a tensor, which results in memory and computation bottlenecks. In this paper, we propose a novel attention mechanism called "Multi-mask Tensorized Self-Attention" (MTSA), which is as fast and as memory-efficient as a CNN, but significantly outperforms previous CNN-/RNN-/attention-based models. MTSA 1) captures both pairwise (token2token) and global (source2token) dependencies by a novel compatibility function composed of dot-product and additive attentions, 2) uses a tensor to represent the feature-wise alignment scores for better expressive power but only requires parallelizable matrix multiplications, and 3) combines multi-head with multi-dimensional attentions, and applies a distinct positional mask to each head (subspace), so the memory and computation can be distributed to multiple heads, each with sequential information encoded independently. The experiments show that a CNN/RNN-free model based on MTSA achieves state-of-the-art or competitive performance on nine NLP benchmarks with compelling memory- and time-efficiency.