HLA: Hadamard Linear Attention

TL;DR

This paper introduces Hadamard Linear Attention (HLA), a novel linear attention method that improves approximation of softmax in transformers, enabling efficient processing of large token sequences in applications like video generation.

Contribution

HLA proposes a new nonlinearity applied after pairwise similarities, resulting in a higher-degree rational approximation of softmax with an efficient computation scheme.

Findings

HLA effectively approximates softmax in large-scale transformer models.

The method reduces computational complexity without tensor reshaping.

Demonstrated success in large diffusion transformer for video generation.

Abstract

The attention mechanism is an important reason for the success of transformers. It relies on computing pairwise relations between tokens. To reduce the high computational cost of standard quadratic attention, linear attention has been proposed as an efficient approximation. It employs kernel functions that are applied independently to the inputs before the pairwise similarities are calculated. That allows for an efficient computational procedure which, however, amounts to a low-degree rational function approximating softmax. We propose Hadamard Linear Attention (HLA). Unlike previous works on linear attention, the nonlinearity in HLA is not applied separately to queries and keys, but, analogously to standard softmax attention, after the pairwise similarities have been computed. It will be shown that the proposed nonlinearity amounts to a higher-degree rational function to approximate softmax. An efficient computational scheme for the proposed method is derived that is similar to that of standard linear attention. In contrast to other approaches, no time-consuming tensor reshaping is necessary to apply the proposed algorithm. The effectiveness of the approach is demonstrated by applying it to a large diffusion transformer model for video generation, an application that involves very large amounts of tokens.