DRESS: Dynamic RESource-reservation Scheme for Congested Data-intensive Computing Platforms

TL;DR

This paper introduces DRESS, a dynamic resource reservation scheduling scheme for data-intensive platforms that improves job completion times by adaptively reserving resources based on job demands and system monitoring.

Contribution

DRESS is a novel scheduling strategy that classifies jobs, reserves resources accordingly, and dynamically adjusts reservations to optimize performance in resource-constrained environments.

Findings

Reduces completion time for targeted job categories by up to 76.1%.

Maintains stable overall system performance.

Adapts resource reservations based on job demand patterns.

Abstract

In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1% in our experiments, and in the meanwhile, maintains a stable overall system performance.