Efficient Runtime Profiling for Black-box Machine Learning Services on Sensor Streams

TL;DR

This paper introduces an efficient black-box runtime profiling method for containerized machine learning services on sensor streams, enabling adaptive resource management in distributed environments.

Contribution

It presents a hardware-agnostic profiling strategy that quickly captures runtime behavior of ML jobs, facilitating resource optimization in streaming scenarios.

Findings

Profiles ML jobs after short initial phase

Captures general runtime behavior effectively

Supports adaptive resource allocation

Abstract

In highly distributed environments such as cloud, edge and fog computing, the application of machine learning for automating and optimizing processes is on the rise. Machine learning jobs are frequently applied in streaming conditions, where models are used to analyze data streams originating from e.g. video streams or sensory data. Often the results for particular data samples need to be provided in time before the arrival of next data. Thus, enough resources must be provided to ensure the just-in-time processing for the specific data stream. This paper focuses on proposing a runtime modeling strategy for containerized machine learning jobs, which enables the optimization and adaptive adjustment of resources per job and component. Our black-box approach assembles multiple techniques into an efficient runtime profiling method, while making no assumptions about underlying hardware, data streams, or applied machine learning jobs. The results show that our method is able to capture the general runtime behaviour of different machine learning jobs already after a short profiling phase.