A Game-Theoretic Approach for Runtime Capacity Allocation in MapReduce

TL;DR

This paper introduces a game-theoretic distributed algorithm for runtime capacity allocation in Hadoop clusters, aiming to reduce power consumption while meeting SLA deadlines and preventing job rejections.

Contribution

It presents a novel game-theoretic approach for dynamic resource allocation in MapReduce, improving efficiency and SLA compliance in shared data centers.

Findings

Significant power savings achieved.

Enhanced SLA adherence demonstrated.

Effective distributed algorithm developed.

Abstract

Nowadays many companies have available large amounts of raw, unstructured data. Among Big Data enabling technologies, a central place is held by the MapReduce framework and, in particular, by its open source implementation, Apache Hadoop. For cost effectiveness considerations, a common approach entails sharing server clusters among multiple users. The underlying infrastructure should provide every user with a fair share of computational resources, ensuring that Service Level Agreements (SLAs) are met and avoiding wastes. In this paper we consider two mathematical programming problems that model the optimal allocation of computational resources in a Hadoop 2.x cluster with the aim to develop new capacity allocation techniques that guarantee better performance in shared data centers. Our goal is to get a substantial reduction of power consumption while respecting the deadlines stated in the SLAs and avoiding penalties associated with job rejections. The core of this approach is a distributed algorithm for runtime capacity allocation, based on Game Theory models and techniques, that mimics the MapReduce dynamics by means of interacting players, namely the central Resource Manager and Class Managers.

Figures

Full text

See pages 1-last of marxiv.pdf