Cache Coherence Over Disaggregated Memory

TL;DR

This paper presents SELCC, a novel cache coherence protocol for disaggregated memory systems that reduces remote computing overhead and improves performance by embedding cache metadata into RDMA operations.

Contribution

It introduces SELCC, a cache coherence protocol that maintains coherence without burdening remote memory servers, aligning with MSI protocol, and embedding metadata into RDMA latch words.

Findings

SELCC outperforms RPC-based protocols in cache coherence efficiency.

Applications on SELCC achieve comparable or better performance than existing disaggregated memory solutions.

SELCC enables effective cache management with minimal remote computing power requirements.

Abstract

Disaggregating memory from compute offers the opportunity to better utilize stranded memory in cloud data centers. It is important to cache data in the compute nodes and maintain cache coherence across multiple compute nodes. However, the limited computing power on disaggregated memory servers makes traditional cache coherence protocols suboptimal, particularly in the case of stranded memory. This paper introduces SELCC; a Shared-Exclusive Latch Cache Coherence protocol that maintains cache coherence without imposing any computational burden on the remote memory side. It aligns the state machine of the shared-exclusive latch protocol with the MSI protocol , thereby ensuring both atomicity of data access and cache coherence with sequential consistency. SELCC embeds cache-ownership metadata directly into the RDMA latch word, enabling efficient cache ownership management via RDMA atomic operations. SELCC can serve as an abstraction layer over disaggregated memory with APIs that resemble main-memory accesses. A concurrent B-tree and three transaction concurrency control algorithms are realized using SELCC's abstraction layer. Experimental results show that SELCC significantly outperforms Remote-Procedure-Call-based protocols for cache coherence under limited remote computing power. Applications on SELCC achieve comparable or superior performance over disaggregated memory compared to competitors.