Pangolin: A Fault-Tolerant Persistent Memory Programming Library

TL;DR

Pangolin is a fault-tolerant library for non-volatile memory that ensures data integrity with minimal overhead, enabling reliable persistent data structures with high performance.

Contribution

It introduces a novel fault-tolerance mechanism combining checksums, parity, and micro-buffering for NVMM, supporting objects of any size with low overhead and online error detection.

Findings

Achieves comparable performance to existing solutions

Uses only 1% storage overhead for gigabyte-scale pools

Provides automatic detection and recovery from data corruption

Abstract

Non-volatile main memory (NVMM) allows programmers to build complex, persistent, pointer-based data structures that can offer substantial performance gains over conventional approaches to managing persistent state. This programming model removes the file system from the critical path which improves performance, but it also places these data structures out of reach of file system-based fault tolerance mechanisms (e.g., block-based checksums or erasure coding). Without fault-tolerance, using NVMM to hold critical data will be much less attractive. This paper presents Pangolin, a fault-tolerant persistent object library designed for NVMM. Pangolin uses a combination of checksums, parity, and micro-buffering to protect an application's objects from both media errors and corruption due to software bugs. It provides these protections for objects of any size and supports automatic, online detection of data corruption and recovery. The required storage overhead is small (1% for gigabyte-sized pools of NVMM). Pangolin provides stronger protection, requires orders of magnitude less storage overhead, and achieves comparable performance relative to the current state-of-the-art fault-tolerant persistent object library.