Metall: A Persistent Memory Allocator For Data-Centric Analytics

TL;DR

Metall is a new persistent memory allocator that improves data structure reuse and performance in data-centric analytics applications by leveraging NVRAM devices and efficient memory management techniques.

Contribution

It introduces Metall, a persistent memory allocator with a high-performance algorithm and C++ interface, enabling efficient data structure persistence across process lifecycles.

Findings

Achieved up to 11.7x performance improvement over Boost.Interprocess.

Achieved up to 48.3x performance improvement over memkind.

Successfully integrated into a graph processing framework, demonstrating adaptability.

Abstract

Data analytics applications transform raw input data into analytics-specific data structures before performing analytics. Unfortunately, such data ingestion step is often more expensive than analytics. In addition, various types of NVRAM devices are already used in many HPC systems today. Such devices will be useful for storing and reusing data structures beyond a single process life cycle. We developed Metall, a persistent memory allocator built on top of the memory-mapped file mechanism. Metall enables applications to transparently allocate custom C++ data structures into various types of persistent memories. Metall incorporates a concise and high-performance memory management algorithm inspired by Supermalloc and the rich C++ interface developed by Boost.Interprocess library. On a dynamic graph construction workload, Metall achieved up to 11.7x and 48.3x performance improvements over Boost.Interprocess and memkind (PMEM kind), respectively. We also demonstrate Metall's high adaptability by integrating Metall into a graph processing framework, GraphBLAS Template Library. This study's outcomes indicate that Metall will be a strong tool for accelerating future large-scale data analytics by allowing applications to leverage persistent memory efficiently.