Implicit Kernel Attention

TL;DR

This paper introduces a generalized implicit kernel attention mechanism that unifies and extends traditional attention by incorporating implicit kernels, $L^{p}$ norms, and structured multi-heads, leading to improved performance across various tasks.

Contribution

It proposes a novel implicit kernel attention framework that generalizes existing attention mechanisms with flexible kernel and norm choices, enhancing model adaptability.

Findings

Improved performance on classification, translation, and regression tasks.

Demonstrated effectiveness of implicit kernel attention over traditional methods.

Extended attention to structured multi-head configurations.

Abstract

\textit{Attention} computes the dependency between representations, and it encourages the model to focus on the important selective features. Attention-based models, such as Transformer and graph attention network (GAT), are widely utilized for sequential data and graph-structured data. This paper suggests a new interpretation and generalized structure of the attention in Transformer and GAT. For the attention in Transformer and GAT, we derive that the attention is a product of two parts: 1) the RBF kernel to measure the similarity of two instances and 2) the exponential of $L^{2}$ norm to compute the importance of individual instances. From this decomposition, we generalize the attention in three ways. First, we propose implicit kernel attention with an implicit kernel function instead of manual kernel selection. Second, we generalize $L^{2}$ norm as the $L^{p}$ norm. Third, we extend our attention to structured multi-head attention. Our generalized attention shows better performance on classification, translation, and regression tasks.