CXL over Ethernet: A Novel FPGA-based Memory Disaggregation Design in Data Centers

TL;DR

This paper introduces a novel FPGA-based memory disaggregation method over Ethernet using CXL and RDMA, enabling low-latency remote memory access across servers and racks in data centers.

Contribution

It proposes a new CXL over Ethernet approach supporting native memory semantics and extending physical range beyond rack level, with prototype implementation and latency optimization.

Findings

Average remote memory access latency is 1.97 μs, 37% lower than industry baseline.

Latency reduced to 415 ns with cache block and hit access on FPGA.

Supports native load/store semantics over Ethernet for memory disaggregation.

Abstract

Memory resources in data centers generally suffer from low utilization and lack of dynamics. Memory disaggregation solves these problems by decoupling CPU and memory, which currently includes approaches based on RDMA or interconnection protocols such as Compute Express Link (CXL). However, the RDMA-based approach involves code refactoring and higher latency. The CXL-based approach supports native memory semantics and overcomes the shortcomings of RDMA, but is limited within rack level. In addition, memory pooling and sharing based on CXL products are currently in the process of early exploration and still take time to be available in the future. In this paper, we propose the CXL over Ethernet approach that the host processor can access the remote memory with memory semantics through Ethernet. Our approach can support native memory load/store access and extends the physical range to cross server and rack levels by taking advantage of CXL and RDMA technologies. We prototype our approach with one server and two FPGA boards with 100 Gbps network and measure the memory access latency. Furthermore, we optimize the memory access path by using data cache and congestion control algorithm in the critical path to further lower access latency. The evaluation results show that the average latency for the server to access remote memory is 1.97 {\mu}s, which is about 37% lower than the baseline latency in the industry. The latency can be further reduced to 415 ns with cache block and hit access on FPGA.