# Exploring the Fairness and Resource Distribution in an Apache Mesos Environment

## TL;DR

This paper investigates resource fairness and distribution in Apache Mesos, analyzing how scheduling policies and parameters affect fairness and cluster utilization through experiments with various frameworks.

## Contribution

It provides an empirical analysis of resource fairness in Mesos, demonstrating how scheduling decisions impact fairness and proposing methods to improve resource allocation.

## Key findings

- Bin-Packing with Marathon reduces unfairness from 38% to 3%.
- Reducing unused resources decreases unfairness from 90% to 28%.
- Adjusting task arrival rates lowers unfairness from 23% to 7%.

## Abstract

Apache Mesos, a cluster-wide resource manager, is widely deployed in massive scale at several Clouds and Data Centers. Mesos aims to provide high cluster utilization via fine grained resource co-scheduling and resource fairness among multiple users through Dominant Resource Fairness (DRF) based allocation. DRF takes into account different resource types (CPU, Memory, Disk I/O) requested by each application and determines the share of each cluster resource that could be allocated to the applications. Mesos has adopted a two-level scheduling policy: (1) DRF to allocate resources to competing frameworks and (2) task level scheduling by each framework for the resources allocated during the previous step. We have conducted experiments in a local Mesos cluster when used with frameworks such as Apache Aurora, Marathon, and our own framework Scylla, to study resource fairness and cluster utilization. Experimental results show how informed decision regarding second level scheduling policy of frameworks and attributes like offer holding period, offer refusal cycle and task arrival rate can reduce unfair resource distribution. Bin-Packing scheduling policy on Scylla with Marathon can reduce unfair allocation from 38\% to 3\%. By reducing unused free resources in offers we bring down the unfairness from to 90\% to 28\%. We also show the effect of task arrival rate to reduce the unfairness from 23\% to 7\%.