ATLAS: An Adaptive Failure-aware Scheduler for Hadoop

TL;DR

ATLAS is an adaptive failure-aware scheduler for Hadoop that predicts and mitigates task failures in cloud environments, significantly improving job success rates and reducing execution time and resource usage.

Contribution

This paper introduces ATLAS, a novel scheduler that dynamically predicts failures and adapts scheduling decisions in Hadoop to enhance reliability and efficiency in cloud settings.

Findings

Reduces failed jobs by up to 28%.

Decreases failed tasks by up to 39%.

Speeds up job completion by 10 minutes on average.

Abstract

Hadoop has become the de facto standard for processing large data in today's cloud environment. The performance of Hadoop in the cloud has a direct impact on many important applications ranging from web analytic, web indexing, image and document processing to high-performance scientific computing. However, because of the scale, complexity and dynamic nature of the cloud, failures are common and these failures often impact the performance of jobs running in Hadoop. Although Hadoop possesses built-in failure detection and recovery mechanisms, several scheduled jobs still fail because of unforeseen events in the cloud environment. A single task failure can cause the failure of the whole job and unpredictable job running times. In this report, we propose ATLAS (AdapTive faiLure-Aware Scheduler), a new scheduler for Hadoop that can adapt its scheduling decisions to events occurring in the cloud environment. Using statistical models, ATLAS predicts task failures and adjusts its scheduling decisions on the fly to reduce task failure occurrences. We implement ATLAS in the Hadoop framework of Amazon Elastic MapReduce (EMR) and perform a case study to compare its performance with those of the FIFO, Fair and Capacity schedulers. Results show that ATLAS can reduce the percentage of failed jobs by up to 28% and the percentage of failed tasks by up to 39%, and the total execution time of jobs by 10 minutes on average. ATLAS also reduces CPU and memory usages.