PipeSim: Trace-driven Simulation of Large-Scale AI Operations Platforms

TL;DR

This paper introduces PipeSim, a trace-driven simulation environment for optimizing large-scale AI operations, enabling tailored operational strategies to improve efficiency and model-specific outcomes.

Contribution

It presents a comprehensive, stochastic simulation model and toolkit based on real IBM AI platform data for evaluating AI workflow management strategies.

Findings

Effective simulation of AI pipeline interactions with infrastructure

Enables testing of scheduling and resource allocation strategies

Supports analysis of AI model metrics like accuracy and fairness

Abstract

Operationalizing AI has become a major endeavor in both research and industry. Automated, operationalized pipelines that manage the AI application lifecycle will form a significant part of tomorrow's infrastructure workloads. To optimize operations of production-grade AI workflow platforms we can leverage existing scheduling approaches, yet it is challenging to fine-tune operational strategies that achieve application-specific cost-benefit tradeoffs while catering to the specific domain characteristics of machine learning (ML) models, such as accuracy, robustness, or fairness. We present a trace-driven simulation-based experimentation and analytics environment that allows researchers and engineers to devise and evaluate such operational strategies for large-scale AI workflow systems. Analytics data from a production-grade AI platform developed at IBM are used to build a comprehensive simulation model. Our simulation model describes the interaction between pipelines and system infrastructure, and how pipeline tasks affect different ML model metrics. We implement the model in a standalone, stochastic, discrete event simulator, and provide a toolkit for running experiments. Synthetic traces are made available for ad-hoc exploration as well as statistical analysis of experiments to test and examine pipeline scheduling, cluster resource allocation, and similar operational mechanisms.