Megha: Decentralized Global Fair Scheduling for Federated Clusters

TL;DR

Megha is a decentralized scheduling framework for federated data center clusters that ensures fair resource allocation with low overhead by using a distributed global scheduler and logical partitioning.

Contribution

It introduces Megha, a novel decentralized resource management system that operates without centralized data stores, improving scalability and fairness in federated clusters.

Findings

Schedules tasks with low resource allocation times (~tens of milliseconds)

Ensures workload fairness and placement constraints

Operates with low scheduling overheads

Abstract

Increasing scale and heterogeneity in data centers have led to the development of federated clusters such as KubeFed, Hydra, and Pigeon, that federate individual data center clusters. In our work, we introduce Megha, a novel decentralized resource management framework for such federated clusters. Megha employs flexible logical partitioning of clusters to distribute its scheduling load, ensuring that the requirements of the workload are satisfied with very low scheduling overheads. It uses a distributed global scheduler that does not rely on a centralized data store but, instead, works with eventual consistency, unlike other schedulers that use a tiered architecture or rely on centralized databases. Our experiments with Megha show that it can schedule tasks taking into account fairness and placement constraints with low resource allocation times - in the order of tens of milliseconds.