Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

TL;DR

Enel is a novel dynamic scaling method for distributed dataflow systems that models jobs with message propagation on attributed graphs to optimize resource allocation amid performance variability.

Contribution

Enel introduces a graph propagation-based approach for dynamic scaling in dataflow systems, considering execution context and task statistics for effective rescaling decisions.

Findings

Successfully reacts to node failures with effective rescaling

Reuses across different execution contexts

Identifies optimal rescaling actions in Spark jobs

Abstract

Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics. While runtime prediction models can be used to initially select appropriate cluster resources given target runtimes, the actual runtime performance of dataflow jobs depends on several factors and varies over time. Yet, in many situations, dynamic scaling can be used to meet formulated runtime targets despite significant performance variance. This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs and, thus, allows for deriving effective rescaling decisions. For this, Enel incorporates descriptive properties that capture the respective execution context, considers statistics from individual dataflow tasks, and propagates predictions through the job graph to eventually find an optimized new scale-out. Our evaluation of Enel with four iterative Spark jobs shows that our approach is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.