A Primal-Dual Framework for Transformers and Neural Networks

TL;DR

This paper introduces a unified primal-dual framework for understanding and designing attention mechanisms in transformers, connecting them to support vector regression, and proposes two novel attention methods that improve efficiency and accuracy.

Contribution

It provides a principled, support vector-based framework for constructing attention layers and introduces two new attention mechanisms, Attention-BN and Attention-SH.

Findings

Attention-BN reduces head redundancy and improves accuracy.

Attention-SH enhances efficiency with less training data.

Both methods outperform traditional attention in practical tasks.

Abstract

Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision. Like neural network layers, these attention mechanisms are often developed by heuristics and experience. To provide a principled framework for constructing attention layers in transformers, we show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem, whose primal formulation has the form of a neural network layer. Using our framework, we derive popular attention layers used in practice and propose two new attentions: 1) the Batch Normalized Attention (Attention-BN) derived from the batch normalization layer and 2) the Attention with Scaled Head (Attention-SH) derived from using less training data to fit the SVR model. We empirically demonstrate the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model's accuracy, and improving the model's efficiency in a variety of practical applications including image and time-series classification.