Trustworthy Scheduling for Big Data Applications

TL;DR

This paper introduces X-Sched, a middleware that enhances containerized big data application scheduling by providing explainable, actionable guidance on resource configurations to meet performance and SLOs.

Contribution

X-Sched uniquely combines explainability techniques with machine learning to improve scheduling transparency and decision-making in containerized environments.

Findings

X-Sched effectively guides resource configuration decisions.

It ensures task execution aligns with performance goals.

Experimental validation shows practical benefits.

Abstract

Recent advances in modern containerized execution environments have resulted in substantial benefits in terms of elasticity and more efficient utilization of computing resources. Although existing schedulers strive to optimize performance metrics like task execution times and resource utilization, they provide limited transparency into their decision-making processes or the specific actions developers must take to meet Service Level Objectives (SLOs). In this work, we propose X-Sched, a middleware that uses explainability techniques to generate actionable guidance on resource configurations that makes task execution in containerized environments feasible, under resource and time constraints. X-Sched addresses this gap by integrating counterfactual explanations with advanced machine learning models, such as Random Forests, to efficiently identify optimal configurations. This approach not only ensures that tasks are executed in line with performance goals but also gives users clear, actionable insights into the rationale behind scheduling decisions. Our experimental results validated with data from real-world execution environments, illustrate the efficiency, benefits and practicality of our approach.