LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms

TL;DR

This paper introduces LLMPrism, a black-box system that uses network flow data to non-intrusively monitor, reconstruct training timelines, and diagnose performance issues in large-scale LLM training platforms, improving efficiency and resource utilization.

Contribution

LLMPrism is the first system to enable non-intrusive, accurate performance diagnosis for production LLM training platforms using network flow data.

Findings

Achieves timeline reconstruction error within 0.3%

Effectively diagnoses various performance issues

Deployed on large-scale production platform since Oct. 2024

Abstract

Large Language Models (LLMs) have brought about revolutionary changes in diverse fields, rendering LLM training of utmost importance for modern enterprises. To meet this demand, multi-tenant large-scale LLM training platforms have been built to offer LLM training services. Nevertheless, due to the complexity and synchronous nature of LLM training process, performance issues occur frequently and can result in substantial resource wastage. The limited visibility from the perspective of platform providers impedes existing profiling methods and poses challenges to the monitoring and diagnosis of the performance of LLM training jobs. For the first time, this paper proposes the utilization of underlying network flow data to reconstruct the training timelines of jobs based on the distinct characteristics in the LLM training procedure. We design LLMPrism, the first black-box performance diagnosis system for LLM training platforms. By progressively recognizing LLM training jobs, identifying their parallelism strategies, and reconstructing the training timelines, LLMPrism achieves non-intrusive, lightweight, and continuous monitoring of LLM training systems. Leveraging this monitoring capability, it further effectively diagnoses potential performance issues. Since Oct. 2024, LLMPrism has been deployed on our large-scale production Platform-X, in which the evaluations and deployment experiences demonstrate that LLMPrism can achieve accurate timeline reconstruction with an error within 0.3% and effectively diagnose various performance issues.