IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference

TL;DR

IntAttention introduces a fully integer, efficient attention pipeline for Transformer models on edge devices, significantly reducing latency and energy consumption without retraining.

Contribution

It presents the first fully integer attention pipeline with IndexSoftmax, eliminating datatype conversions and boosting edge inference efficiency.

Findings

Up to 3.7x speedup over FP16 baselines

61% energy reduction compared to FP16

2.0x faster than conventional INT8 attention pipelines

Abstract

Deploying Transformer models on edge devices is limited by latency and energy budgets. While INT8 quantization effectively accelerates the primary matrix multiplications, it exposes the softmax as the dominant bottleneck. This stage incurs a costly dequantize-softmax-requantize detour, which can account for up to 65% of total attention latency and disrupts the end-to-end integer dataflow critical for edge hardware efficiency. To address this limitation, we present IntAttention, the first fully integer, plug-and-play attention pipeline without retraining. At the core of our approach lies IndexSoftmax, a hardware-friendly operator that replaces floating-point exponentials entirely within the integer domain. IntAttention integrates sparsity-aware clipping, a 32-entry lookup-table approximation, and direct integer normalization, thereby eliminating all datatype conversion overhead. We evaluate IntAttention and demonstrate consistent and substantial gains. Our method achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines and 2.0x faster than conventional INT8 attention pipelines on Armv8 CPUs. These gains are achieved with high-fidelity accuracy comparable to baselines across diverse language and vision models, enabling practical and efficient Transformer inference on commodity edge devices. Code will be released in later version of this work.