Cost-Effective Attention Mechanisms for Low Resource Settings: Necessity & Sufficiency of Linear Transformations

TL;DR

This paper introduces simplified attention mechanisms that reduce computational costs and parameters in deep learning models, achieving comparable or better performance in NLP and vision tasks, especially in low-resource settings.

Contribution

It proposes three novel attention variants with fewer linear transformations, significantly reducing model size and computational requirements while maintaining or improving performance.

Findings

Models are 25-50% lighter than standard SDPA.

Performance degradation is negligible with size reduction.

Super Attention outperforms SDPA by up to 10%.

Abstract

From natural language processing to vision, Scaled Dot Product Attention (SDPA) is the backbone of most modern deep learning applications. Unfortunately, its memory and computational requirements can be prohibitive in low-resource settings. In this paper, we improve its efficiency without sacrificing its versatility. We propose three attention variants where we remove consecutive linear transformations or add a novel one, and evaluate them on a range of standard NLP and vision tasks. Our proposed models are substantially lighter than standard SDPA (and have 25-50% fewer parameters). We show that the performance cost of these changes is negligible relative to size reduction and that in one case (Super Attention) we succeed in outperforming SDPA by up to 10% while improving its speed and reducing its parameters by 25%.