CXL Memory as Persistent Memory for Disaggregated HPC: A Practical Approach

TL;DR

This paper investigates the use of CXL memory as a practical persistent memory solution for disaggregated HPC systems, demonstrating its performance and ease of programming model transition through physical experiments and benchmarks.

Contribution

It provides the first comprehensive experimental evaluation of CXL memory as persistent memory in HPC, including a prototype implementation and performance benchmarking.

Findings

CXL memory can emulate PMem characteristics in HPC environments.

CXL-DDR4 modules achieve bandwidth comparable to local DDR4 memory.

Transitioning from traditional PMem programming models to CXL is straightforward.

Abstract

In the landscape of High-Performance Computing (HPC), the quest for efficient and scalable memory solutions remains paramount. The advent of Compute Express Link (CXL) introduces a promising avenue with its potential to function as a Persistent Memory (PMem) solution in the context of disaggregated HPC systems. This paper presents a comprehensive exploration of CXL memory's viability as a candidate for PMem, supported by physical experiments conducted on cutting-edge multi-NUMA nodes equipped with CXL-attached memory prototypes. Our study not only benchmarks the performance of CXL memory but also illustrates the seamless transition from traditional PMem programming models to CXL, reinforcing its practicality. To substantiate our claims, we establish a tangible CXL prototype using an FPGA card embodying CXL 1.1/2.0 compliant endpoint designs (Intel FPGA CXL IP). Performance evaluations, executed through the STREAM and STREAM-PMem benchmarks, showcase CXL memory's ability to mirror PMem characteristics in App-Direct and Memory Mode while achieving impressive bandwidth metrics with Intel 4th generation Xeon (Sapphire Rapids) processors. The results elucidate the feasibility of CXL memory as a persistent memory solution, outperforming previously established benchmarks. In contrast to published DCPMM results, our CXL-DDR4 memory module offers comparable bandwidth to local DDR4 memory configurations, albeit with a moderate decrease in performance. The modified STREAM-PMem application underscores the ease of transitioning programming models from PMem to CXL, thus underscoring the practicality of adopting CXL memory.