Two stage cluster for resource optimization with Apache Mesos

TL;DR

This paper introduces a two-stage clustering approach using a small initial cluster to accurately estimate job resource needs, leading to improved resource utilization and reduced wait times in large clusters.

Contribution

The paper presents a novel two-stage clustering method that enhances resource estimation accuracy and utilization in cluster management systems.

Findings

Resource estimation accuracy of 90% for memory and 94% for CPU.

Memory utilization improved by 22%, CPU by 53%.

Better throughput and resource utilization compared to default methods.

Abstract

As resource estimation for jobs is difficult, users often overestimate their requirements. Both commercial clouds and academic campus clusters suffer from low resource utilization and long wait times as the resource estimates for jobs, provided by users, is inaccurate. We present an approach to statistically estimate the actual resource requirement of a job in a Little cluster before the run in a Big cluster. The initial estimation on the little cluster gives us a view of how much actual resources a job requires. This initial estimate allows us to accurately allocate resources for the pending jobs in the queue and thereby improve throughput and resource utilization. In our experiments, we determined resource utilization estimates with an average accuracy of 90% for memory and 94% for CPU, while we make better utilization of memory by an average of 22% and CPU by 53%, compared to the default job submission methods on Apache Aurora and Apache Mesos.