Learning Interpretable Scheduling Algorithms for Data Processing Clusters

TL;DR

This paper introduces an interpretable, distilled scheduling policy for data processing clusters that mimics complex deep learning models, offering better adaptability and transparency over traditional RL approaches.

Contribution

We propose a novel distillation method to create simple, interpretable scheduling policies that replicate deep learning models' decisions and improve adaptability in unseen workloads.

Findings

High fidelity to deep learning models' decisions

Outperforms complex models with added heuristics

Eases adaptation to edge cases

Abstract

Workloads in data processing clusters are often represented in the form of DAG (Directed Acyclic Graph) jobs. Scheduling DAG jobs is challenging. Simple heuristic scheduling algorithms are often adopted in practice in production data centres. There is much room for scheduling performance optimisation for cost saving. Recently, reinforcement learning approaches (like decima) have been attempted to optimise DAG job scheduling and demonstrate clear performance gain in comparison to traditional algorithms. However, reinforcement learning (RL) approaches face their own problems in real-world deployment. In particular, their black-box decision making processes and generalizability in unseen workloads may add a non-trivial burden to the cluster administrators. Moreover, adapting RL models on unseen workloads often requires significant amount of training data, which leaves edge cases run in a sub-optimal mode. To fill the gap, we propose a new method to distill a simple scheduling policy based on observations of the behaviours of a complex deep learning model. The simple model not only provides interpretability of scheduling decisions, but also adaptive to edge cases easily through tuning. We show that our method achieves high fidelity to the decisions made by deep learning models and outperforms these models when additional heuristics are taken into account.