Scheduler Technologies in Support of High Performance Data Analysis

TL;DR

This paper compares HPC and Big Data schedulers, highlighting their features, performance differences, and the impact of multilevel scheduling on job latency and resource utilization in scalable computing environments.

Contribution

It provides a comparative analysis of HPC and Big Data schedulers and evaluates the effectiveness of multilevel scheduling in improving utilization and reducing job latency.

Findings

All schedulers show low utilization for short jobs.

Multilevel scheduling improves resource utilization.

Job launch benchmarking reveals performance differences.

Abstract

Job schedulers are a key component of scalable computing infrastructures. They orchestrate all of the work executed on the computing infrastructure and directly impact the effectiveness of the system. Recently, job workloads have diversified from long-running, synchronously-parallel simulations to include short-duration, independently parallel high performance data analysis (HPDA) jobs. Each of these job types requires different features and scheduler tuning to run efficiently. A number of schedulers have been developed to address both job workload and computing system heterogeneity. High performance computing (HPC) schedulers were designed to schedule large-scale scientific modeling and simulations on supercomputers. Big Data schedulers were designed to schedule data processing and analytic jobs on clusters. This paper compares and contrasts the features of HPC and Big Data schedulers with a focus on accommodating both scientific computing and high performance data analytic workloads. Job latency is critical for the efficient utilization of scalable computing infrastructures, and this paper presents the results of job launch benchmarking of several current schedulers: Slurm, Son of Grid Engine, Mesos, and Yarn. We find that all of these schedulers have low utilization for short-running jobs. Furthermore, employing multilevel scheduling significantly improves the utilization across all schedulers.