eACGM: Non-instrumented Performance Tracing and Anomaly Detection towards Machine Learning Systems

TL;DR

eACGM is a comprehensive, non-intrusive system monitoring framework for AI/ML that uses eBPF and statistical modeling to detect performance anomalies across hardware and software components in real-time.

Contribution

It introduces a full-stack, code-instrumentation-free monitoring system that combines hardware and software performance data with GMM-based anomaly detection for AI/ML systems.

Findings

Successfully detects latency anomalies and hardware failures.

Maintains low overhead during distributed training.

Proves effective in real-world production environments.

Abstract

We present eACGM, a full-stack AI/ML system monitoring framework based on eBPF. eACGM collects real-time performance data from key hardware components, including the GPU and network communication layer, as well as from key software stacks such as CUDA, Python, and PyTorch, all without requiring any code instrumentation or modifications. Additionally, it leverages libnvml to gather process-level GPU resource usage information. By applying a Gaussian Mixture Model (GMM) to the collected multidimensional performance metrics for statistical modeling and clustering analysis, eACGM effectively identifies complex failure modes, such as latency anomalies, hardware failures, and communication inefficiencies, enabling rapid diagnosis of system bottlenecks and abnormal behaviors. To evaluate eACGM's effectiveness and practicality, we conducted extensive empirical studies and case analyses in multi-node distributed training scenarios. The results demonstrate that eACGM, while maintaining a non-intrusive and low-overhead profile, successfully captures critical performance anomalies during model training and inference. Its stable anomaly detection performance and comprehensive monitoring capabilities validate its applicability and scalability in real-world production environments, providing strong support for performance optimization and fault diagnosis in large-scale AI/ML systems.