Peacock: Probe-Based Scheduling of Jobs by Rotating Between Elastic Queues

TL;DR

Peacock is a distributed probe-based scheduler that improves job scheduling in data analytics frameworks by rotating probes among elastic queues, reducing latency and head-of-line blocking.

Contribution

It introduces a novel probe rotation technique and probe reordering algorithm to enhance scheduling efficiency over existing solutions.

Findings

Outperforms state-of-the-art schedulers in various loads and cluster sizes.

Reduces head-of-line blocking more effectively than previous methods.

Demonstrates consistent performance improvements in both simulation and real experiments.

Abstract

In this paper, we propose Peacock, a new distributed probe-based scheduler which handles heterogeneous workloads in data analytics frameworks with low latency. Peacock mitigates the \emph{Head-of-Line blocking} problem, i.e., shorter tasks are enqueued behind the longer tasks, better than the state-of-the-art. To this end, we introduce a novel probe rotation technique. Workers form a ring overlay network and rotate probes using elastic queues. It is augmented by a novel probe reordering algorithm executed in workers. We evaluate the performance of Peacock against two state-of-the-art probe-based solutions through both trace-driven simulation and distributed experiment in Spark under various loads and cluster sizes. Our large-scale performance results indicate that Peacock outperforms the state-of-the-art in all cluster sizes and loads. Our distributed experiments confirm our simulation results.