ReduceFormer: Attention with Tensor Reduction by Summation

TL;DR

ReduceFormer is an efficient transformer variant that simplifies attention using reduction and element-wise operations, significantly reducing latency and increasing throughput while maintaining competitive accuracy, ideal for resource-constrained and high-throughput environments.

Contribution

Introduce ReduceFormer, a transformer model that replaces complex attention operations with simple reductions and multiplications, enhancing efficiency without sacrificing accuracy.

Findings

Up to 37% reduction in latency

44% improvement in throughput

Maintains competitive accuracy

Abstract

Transformers have excelled in many tasks including vision. However, efficient deployment of transformer models in low-latency or high-throughput applications is hindered by the computation in the attention mechanism which involves expensive operations such as matrix multiplication and Softmax. To address this, we introduce ReduceFormer, a family of models optimized for efficiency with the spirit of attention. ReduceFormer leverages only simple operations such as reduction and element-wise multiplication, leading to greatly simplified architecture and improved inference performance, with up to 37% reduction in latency and 44% improvement in throughput, while maintaining competitive accuracy comparable to other recent methods. The proposed model family is suitable for edge devices where compute resource and memory bandwidth are limited, as well as for cloud computing where high throughput is sought after.