System-performance and cost modeling of Large Language Model training and inference

TL;DR

This paper presents a comprehensive performance and cost modeling framework for large language model training and inference, incorporating recent innovations and system design considerations to guide future hardware and software development.

Contribution

It introduces an integrated analytical modeling approach that combines recent compute, memory, and communication techniques with cost analysis for LLM systems.

Findings

The model effectively analyzes performance-cost trade-offs across different system architectures.

Incorporates recent innovations like flash attention and mixture of experts into performance modeling.

Provides insights for optimizing hardware-software co-design for LLMs.

Abstract

Large language models (LLMs), based on transformer architectures, have revolutionized numerous domains within artificial intelligence, science, and engineering due to their exceptional scalability and adaptability. However, the exponential growth in LLM size and complexity has outpaced advancements in compute capacity, memory bandwidth, network performance, and cost efficiency, posing significant challenges to their scalability on distributed systems. To address these limitations, alternative model architectures, optimization strategies, communication-aware network topologies, and novel system design approaches have been proposed in literature. This paper introduces a performance-cost modeling methodology for LLM training and inference that integrates state-of-the-art compute techniques with memory optimizations, and latest communication techniques. Building on an analytical performance model, our approach incorporates recent innovations such as the flash attention technique and mixture of experts models to address the memory bandwidth and compute bottlenecks. It also considers the impact of different network topologies and topology-specific communication algorithms with 5D parallellism. The framework also integrates a chiplet cost model. The proposed modeling methodology provides valuable insights to guide future compute system design and facilitates hardware-software co-development, in particular due to its ability to analyze performance-cost trade-offs for various system architectural configurations.