A Fresh Look at the Reliability of Long-term Digital Storage

TL;DR

This paper emphasizes the importance of re-evaluating long-term digital storage system designs by considering diverse failure threats and proposes strategies like rapid fault detection, automated repair, and increased data independence to improve reliability.

Contribution

It introduces a simple failure model for long-term storage and highlights key strategies to enhance reliability, considering hardware, software, and organizational faults.

Findings

Rapid fault detection is crucial for reliability.

Automated repair reduces costs and downtime.

Increasing data replica independence improves fault tolerance.

Abstract

Many emerging Web services, such as email, photo sharing, and web site archives, need to preserve large amounts of quickly-accessible data indefinitely into the future. In this paper, we make the case that these applications' demands on large scale storage systems over long time horizons require us to re-evaluate traditional storage system designs. We examine threats to long-lived data from an end-to-end perspective, taking into account not just hardware and software faults but also faults due to humans and organizations. We present a simple model of long-term storage failures that helps us reason about the various strategies for addressing these threats in a cost-effective manner. Using this model we show that the most important strategies for increasing the reliability of long-term storage are detecting latent faults quickly, automating fault repair to make it faster and cheaper, and increasing the independence of data replicas.