Run Time Prediction for Big Data Iterative ML Algorithms: a KMeans case study

TL;DR

This paper investigates the run time prediction of iterative machine learning algorithms on big data systems, focusing on a case study of K-Means in Spark, to improve understanding and management of computational costs.

Contribution

It provides a statistical model of K-Means run time on Spark, addressing the challenge of predicting execution duration for big data ML algorithms.

Findings

Statistical characterization of K-Means run time

Modeling approach for run time prediction

Insights into big data iterative algorithm performance

Abstract

Data science and machine learning algorithms running on big data infrastructure are increasingly important in activities ranging from business intelligence and analytics to cybersecurity, smart city management, and many fields of science and engineering. As these algorithms are further integrated into daily operations, understanding how long they take to run on a big data infrastructure is paramount to controlling costs and delivery times. In this paper we discuss the issues involved in understanding the run time of iterative machine learning algorithms and provide a case study of such an algorithm - including a statistical characterization and model of the run time of an implementation of K-Means for the Spark big data engine using the Edward probabilistic programming language.