H-FA: A Hybrid Floating-Point and Logarithmic Approach to Hardware Accelerated FlashAttention

TL;DR

H-FA introduces a hybrid floating-point and logarithmic approach to hardware-accelerated FlashAttention, significantly reducing area and power consumption while maintaining performance in transformer attention computations.

Contribution

It proposes a novel hybrid computation method combining floating-point and fixed-point logarithmic representations for efficient hardware implementation of FlashAttention.

Findings

Achieves 26.5% area reduction in hardware

Reduces power consumption by 23.4%

Maintains performance comparable to existing architectures

Abstract

Transformers have significantly advanced AI and machine learning through their powerful attention mechanism. However, computing attention on long sequences can become a computational bottleneck. FlashAttention mitigates this by fusing the softmax and matrix operations into a tiled computation pattern that decouples performance from sequence length. Though designed for GPUs, its simplicity also makes it well suited for direct hardware acceleration. To improve hardware implementation, we compute FlashAttention using a mixture of floating-point and fixed-point logarithm domain representations. Floating-point is used to compute attention scores from query and key matrices, while logarithmic computation simplifies the fused computation of softmax normalization and the multiplication with the value matrix. This transformation, called H-FA, replaces vector-wide floating-point multiplication and division operations by additions and subtractions implemented efficiently with fixed-point arithmetic in the logarithm domain. Exponential function evaluations are effectively omitted and fused with the rest operations, and the final result is directly returned to floating-point arithmetic without any additional hardware overhead. Hardware implementation results at 28nm demonstrate that H-FA achieves a 26.5% reduction in area and a 23.4% reduction in power, on average, compared to FlashAttention parallel hardware architectures built solely with floating-point datapaths, without hindering performance.