Sequence-Aware Split Heuristic to Mitigate SM Underutilization in FlashAttention-3 Low-Head-Count Decoding

TL;DR

This paper introduces a sequence-aware split heuristic for FlashAttention-3 that enhances GPU utilization and decoder efficiency in low-head-count decoding scenarios by enabling sequence-level parallelism.

Contribution

It proposes a novel sequence-aware split policy that mitigates GPU underutilization in low-head-count decoding, improving efficiency without regressions.

Findings

Achieves 21-24% improvement in decoder kernel efficiency.

Addresses GPU occupancy bottleneck in low-head-count regimes.

No regressions observed with the new heuristic.

Abstract

The standard FlashAttention-3 heuristic exhibits a GPU occupancy bottleneck in low-head-count decoding configurations because it disables sequence splitting based on sequence length alone, underutilizing the Streaming Multiprocessors of Hopper GPUs. Our proposed sequence-aware split policy mitigates this by allowing sequence-level parallelism in low-head-count regimes, improving hardware utilization to deliver roughly a 21 to 24% improvement in decoder kernel efficiency on metadata-enabled inference paths, with no observed regressions.