Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling

TL;DR

This paper evaluates the resource efficiency of autoscaling in big data systems like Spark, finding no significant gains over static allocations due to inherent limitations in autoscaling approaches.

Contribution

It provides a conceptual and experimental analysis of autoscaling resource efficiency in Spark, highlighting fundamental limitations.

Findings

No significant resource efficiency gain over static allocations

Inelasticity of node size limits autoscaling benefits

Memory to CPU ratio inautoscaling is inherently constrained

Abstract

Distributed dataflow systems like Spark and Flink enable data-parallel processing of large datasets on clusters. Yet, selecting appropriate computational resources for dataflow jobs is often challenging. For efficient execution, individual resource allocations, such as memory and CPU cores, must meet the specific resource requirements of the job. An alternative to selecting a static resource allocation for a job execution is autoscaling as implemented for example by Spark. In this paper, we evaluate the resource efficiency of autoscaling batch data processing jobs based on resource demand both conceptually and experimentally by analyzing a new dataset of Spark job executions on Google Dataproc Serverless. In our experimental evaluation, we show that there is no significant resource efficiency gain over static resource allocations. We found that the inherent conceptual limitations of such autoscaling approaches are the inelasticity of node size as well as the inelasticity of the ratio of memory to CPU cores.