EcoTransformer: Attention without Multiplication

TL;DR

EcoTransformer introduces a convolution-based attention mechanism that eliminates matrix multiplication, reducing energy consumption while maintaining or improving performance across NLP, bioinformatics, and vision tasks.

Contribution

It proposes a novel attention method using Laplacian kernel convolution with L1 distance, avoiding multiplication and lowering energy costs.

Findings

Performs on par or better than traditional attention in multiple tasks

Consumes significantly less energy

Supersedes previous version with improved design

Abstract

The Transformer, with its scaled dot-product attention mechanism, has become a foundational architecture in modern AI. However, this mechanism is computationally intensive and incurs substantial energy costs. We propose a new Transformer architecture EcoTransformer, in which the output context vector is constructed as the convolution of the values using a Laplacian kernel, where the distances are measured by the L1 metric between the queries and keys. Compared to dot-product based attention, the new attention score calculation is free of matrix multiplication. It performs on par with, or even surpasses, scaled dot-product attention in NLP, bioinformatics, and vision tasks, while consuming significantly less energy. (This version (v2) supersedes v1 and reflects the intended release and licensing.)