Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency

TL;DR

Chopper is a comprehensive GPU profiling framework that reveals key bottlenecks and inefficiencies in large language model training on AMD GPUs, providing insights for optimization and future system design.

Contribution

We introduce Chopper, a novel multi-granularity profiling tool for GPU-based LLM training, and provide the first detailed characterization of Llama 3 8B training on AMD Instinct GPUs.

Findings

Memory determinism enables higher GPU and memory frequencies.

Frequency overhead (DVFS effects) is the largest source of inefficiency.

Identified bottlenecks inform optimization and hardware design improvements.

Abstract

Training large language models (LLMs) efficiently requires a deep understanding of how modern GPU systems behave under real-world distributed training workloads. While prior work has focused primarily on kernel-level performance or single-GPU microbenchmarks, the complex interaction between communication, computation, memory behavior, and power management in multi-GPU LLM training remains poorly characterized. In this work, we introduce Chopper, a profiling and analysis framework that collects, aligns, and visualizes GPU kernel traces and hardware performance counters across multiple granularities (i.e., from individual kernels to operations, layers, phases, iterations, and GPUs). Using Chopper, we perform a comprehensive end-to-end characterization of Llama 3 8B training under fully sharded data parallelism (FSDP) on an eight-GPU AMD InstinctTM MI300X node. Our analysis reveals several previously underexplored bottlenecks and behaviors, such as memory determinism enabling higher, more stable GPU and memory frequencies. We identify several sources of inefficiencies, with frequency overhead (DVFS effects) being the single largest contributor to the gap between theoretical and observed performance, exceeding the impact of MFMA utilization loss, communication/computation overlap, and kernel launch overheads. Overall, Chopper provides the first holistic, multi-granularity characterization of LLM training on AMD InstinctTM MI300X GPUs, yielding actionable insights for optimizing training frameworks, improving power-management strategies, and guiding future GPU architecture and system design.