First CE Matters: On the Importance of Long Term Properties on Memory Failure Prediction

TL;DR

This paper emphasizes the importance of long-term analysis of correctable errors in memory failure prediction, demonstrating that incorporating long-range properties improves early failure detection in data centers.

Contribution

It introduces novel incremental features capturing long-term CE evolution, enhancing machine learning models for memory failure prediction over traditional short-term analysis.

Findings

Predicts memory failures three hours in advance

Improves precision by 21% and recall by 19%

Validated on real-world data from a large cloud provider

Abstract

Dynamic random access memory failures are a threat to the reliability of data centres as they lead to data loss and system crashes. Timely predictions of memory failures allow for taking preventive measures such as server migration and memory replacement. Thereby, memory failure prediction prevents failures from externalizing, and it is a vital task to improve system reliability. In this paper, we revisited the problem of memory failure prediction. We analyzed the correctable errors (CEs) from hardware logs as indicators for a degraded memory state. As memories do not always work with full occupancy, access to faulty memory parts is time distributed. Following this intuition, we observed that important properties for memory failure prediction are distributed through long time intervals. In contrast, related studies, to fit practical constraints, frequently only analyze the CEs from the last fixed-size time interval while ignoring the predating information. Motivated by the observed discrepancy, we study the impact of including the overall (long-range) CE evolution and propose novel features that are calculated incrementally to preserve long-range properties. By coupling the extracted features with machine learning methods, we learn a predictive model to anticipate upcoming failures three hours in advance while improving the average relative precision and recall for 21% and 19% accordingly. We evaluated our methodology on real-world memory failures from the server fleet of a large cloud provider, justifying its validity and practicality.