A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM

TL;DR

PrismLLM enables faithful large-scale LLM training emulation using only a few GPUs, accurately reproducing performance and memory behaviors of extensive GPU clusters.

Contribution

PrismLLM introduces a high-fidelity emulation framework that decouples large-scale execution from physical cluster size, facilitating efficient debugging and optimization.

Findings

Achieves 0.58% average error in iteration time

Less than 0.01% error in peak GPU memory usage

Emulates clusters of up to 8192 GPUs with fewer than 1% of the physical GPUs

Abstract

Large language model (LLM) training today runs on clusters spanning thousands of GPUs. While this scale enables rapid model advances, developing, debugging, and performance-tuning the training framework inevitably becomes complex and costly. This is because engineers often need to reproduce production behaviors to diagnose failures or evaluate optimizations, thereby demanding frequent and even exclusive access to production-scale clusters -- which becomes increasingly hard given that the majority of GPUs are already committed to production workloads. Simulation relies on complex performance models that are difficult to maintain, and downscaled experiments often fail to capture scale-dependent behaviors. We present PrismLLM to decouple large-scale execution from the need to access large clusters, enabling engineers to run and observe ranks of interest under faithful large-scale behavior using only a few GPUs. PrismLLM constructs a high-fidelity execution graph via a slicing-based approach that captures computation, communication, and dependencies of the target scale. Then, PrismLLM performs hybrid emulation where selected ranks execute the original program while the remaining ranks are replayed as virtual participants. Experiments on large-scale LLM training workloads show that PrismLLM accurately reproduces performance and memory behavior, achieving only 0.58\% average error in iteration time and less than 0.01\% error in peak GPU memory usage. PrismLLM can emulate clusters of up to 8192 GPUs using fewer than 1\% of the physical GPUs required by the original deployment.