Justin: Hybrid CPU/Memory Elastic Scaling for Distributed Stream Processing

TL;DR

Justin introduces a hybrid CPU/memory auto-scaler for distributed stream processing that optimizes resource allocation by monitoring operator performance and adjusting resources dynamically, leading to efficient scaling and resource use.

Contribution

Justin is the first auto-scaler to enable hybrid CPU and memory scaling for DSP operators, improving resource efficiency and reconfiguration speed.

Findings

Justin reduces resource usage while maintaining target processing rates.

Justin achieves comparable or fewer reconfiguration steps than existing auto-scalers.

Justin supports fine-grain memory allocation based on operator needs.

Abstract

Distributed Stream Processing (DSP) engines analyze continuous data via queries expressed as a graph of operators. Auto-scalers adjust the number of parallel instances of these operators to support a target rate. Current auto-scalers couple CPU and memory scaling, allocating resources as one-size-fits-all packages. This contrasts with operators' high diversity of requirements. We present Justin, an auto-scaler that enables hybrid CPU and memory scaling of DSP operators. Justin monitors both CPU usage and the performance of operators' storage operations. Its mechanisms enable finegrain memory allocation for tasks upon a query reconfiguration. The Justin policy identifies individual operators' memory pressure and decides between adjusting parallelism and/or memory assignment. We implement Justin in Apache Flink, extending the Flink Kubernetes Operator and the DS2 CPU-only auto-scaler. Using the Nexmark benchmark, our evaluation shows that Justin identifies suitable resource allocation in as many or fewer reconfiguration steps as DS2 and supports a target rate with significantly fewer CPU and memory resources.