Impact of Limpware on HDFS: A Probabilistic Estimation

TL;DR

This paper investigates how hardware degradation, termed limpware, affects HDFS performance variability, providing a probabilistic estimation of its occurrence and impact in large-scale cloud environments.

Contribution

It introduces a probabilistic model to quantify the frequency and impact of limpware in HDFS, addressing a previously overlooked source of performance variability.

Findings

Limpware significantly affects HDFS performance in large-scale systems.

The probabilistic model estimates the occurrence rate of limpware scenarios.

Limpware can cause severe performance degradation in cloud storage systems.

Abstract

With the advent of cloud computing, thousands of machines are connected and managed collectively. This era is confronted with a new challenge: performance variability, primarily caused by large-scale management issues such as hardware failures, software bugs, and configuration mistakes. In our previous work we highlighted one overlooked cause: limpware - hardware whose performance degrades significantly compared to its specification. We showed that limpware can cause severe impact in current scale-out systems. In this report, we quantify how often these scenarios happen in Hadoop Distributed File System.