DAPE V2: Process Attention Score as Feature Map for Length Extrapolation

TL;DR

This paper reinterprets Transformer attention as a feature map and applies convolution to improve length extrapolation, leading to significant performance gains and revealing new avenues for model evolution.

Contribution

It introduces a novel perspective of viewing attention as a feature map and applies convolution to address length extrapolation issues in Transformers.

Findings

Enhanced Transformer performance with convolution-based attention processing

Identified length extrapolation as a feature map problem

Potential for further evolution of Transformer architectures

Abstract

The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens, in contrast to earlier feed-forward neural networks. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers. In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution. Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance.