StraightLine: An End-to-End Resource-Aware Scheduler for Machine Learning Application Requests

TL;DR

StraightLine is an innovative scheduler that optimally allocates resources for ML applications across diverse infrastructure types, reducing response time and failure rates during deployment.

Contribution

It introduces an empirical dynamic placement algorithm for end-to-end resource-aware scheduling in hybrid ML deployment environments.

Findings

Reduces response time for ML deployment requests.

Decreases failure rate in heterogeneous infrastructure.

Improves resource utilization efficiency.

Abstract

The life cycle of machine learning (ML) applications consists of two stages: model development and model deployment. However, traditional ML systems (e.g., training-specific or inference-specific systems) focus on one particular stage or phase of the life cycle of ML applications. These systems often aim at optimizing model training or accelerating model inference, and they frequently assume homogeneous infrastructure, which may not always reflect real-world scenarios that include cloud data centers, local servers, containers, and serverless platforms. We present StraightLine, an end-to-end resource-aware scheduler that schedules the optimal resources (e.g., container, virtual machine, or serverless) for different ML application requests in a hybrid infrastructure. The key innovation is an empirical dynamic placing algorithm that intelligently places requests based on their unique characteristics (e.g., request frequency, input data size, and data distribution). In contrast to existing ML systems, StraightLine offers end-to-end resource-aware placement, thereby it can significantly reduce response time and failure rate for model deployment when facing different computing resources in the hybrid infrastructure.