A Survey of End-to-End Modeling for Distributed DNN Training: Workloads, Simulators, and TCO

TL;DR

This survey reviews the landscape of distributed DNN training simulators, focusing on workload representation, simulation infrastructure, and TCO models, highlighting trends, limitations, and future research directions.

Contribution

It provides a comprehensive overview of existing simulators and models for distributed DNN training, emphasizing workload abstraction, simulation frameworks, and TCO/emissions analysis.

Findings

Comparison of simulation frameworks and TCO models

Identification of common limitations in current tools

Highlighting emerging trends and open challenges

Abstract

Distributed deep neural networks (DNNs) have become a cornerstone for scaling machine learning to meet the demands of increasingly complex applications. However, the rapid growth in model complexity far outpaces CMOS technology scaling, making sustainable and efficient system design a critical challenge. Addressing this requires coordinated co-design across software, hardware, and technology layers. Due to the prohibitive cost and complexity of deploying full-scale training systems, simulators play a pivotal role in enabling this design exploration. This survey reviews the landscape of distributed DNN training simulators, focusing on three major dimensions: workload representation, simulation infrastructure, and models for total cost of ownership (TCO) including carbon emissions. It covers how workloads are abstracted and used in simulation, outlines common workload representation methods, and includes comprehensive comparison tables covering both simulation frameworks and TCO/emissions models, detailing their capabilities, assumptions, and areas of focus. In addition to synthesizing existing tools, the survey highlights emerging trends, common limitations, and open research challenges across the stack. By providing a structured overview, this work supports informed decision-making in the design and evaluation of distributed training systems.