Sim-FA: A GPGPU Simulator Framework for Fine-Grained FlashAttention Pipeline Analysis

TL;DR

This paper introduces Sim-FA, a GPGPU simulation framework for analyzing the FlashAttention pipeline, supporting new GPU features and providing accurate workload modeling for LLMs.

Contribution

It presents a cycle-accurate simulator integrated with FlashAttention-3, supporting recent GPU features and offering improved accuracy over existing models.

Findings

Simulator achieves 5.7% MAPE against H800.

Provides a theoretical analysis explaining inaccuracies in existing models.

Supports new NVIDIA GPU features like Tensor Memory Accelerator.

Abstract

To efficiently support Large Language Models (LLMs), modern GPGPU architectures have introduced new features and programming paradigms, such as warp specialization. These features enable temporal overlap between the producer and consumer, as well as between matrix multiplication and activation function operations, substantially improving performance. To conduct effective AI infrastructure and computer architecture research, cycle-accurate simulators that support these new features, together with analytical models that faithfully capture workload characteristics, are essential. However, existing academic tools provide limited support for these emerging requirements. Existing cycle-accurate simulators do not incorporate new NVIDIA GPU features, such as the Tensor Memory Accelerator (TMA), in a timely manner. Moreover, existing analytical models can misestimate DRAM traffic under certain configurations. In this paper, we build a simulation pipeline from FlashAttention-3 kernel instrumentation to cycle-accurate simulation. The simulator achieves a mean absolute percentage error (MAPE) of 5.7\% and a maximum absolute percentage error of 12.7\% against H800. We also provide a theoretical analysis of FlashAttention-3 and explain why existing analytical models can produce inaccurate traffic estimates.