Pooling Engram Conditional Memory in Large Language Models using CXL

TL;DR

This paper introduces a CXL-based memory pooling approach for Engram in large language models, enabling scalable, low-latency, and cost-efficient storage that maintains high inference performance.

Contribution

It proposes using CXL memory pools for Engram storage in LLMs, improving scalability and efficiency over traditional RDMA-based methods.

Findings

CXL memory pool achieves near-DRAM performance.

Enhanced scalability and cost-efficiency for Engram storage.

Maintains inference performance with CXL integration.

Abstract

Engram conditional memory has emerged as a promising component for LLMs by decoupling static knowledge lookup from dynamic computation. Since Engram exhibits sparse access patterns and supports prefetching, its massive embedding tables are well-suited for offloading to lower-tier memory. In this paper, we propose using Compute Express Link (CXL) memory pool for Engram storage. Compared to RDMA, CXL provides fine-grained and low-latency access required by minimal and discrete retrieval patterns of Engram. We integrate the CXL-based Engram pool into SGLang, achieving near-DRAM end-to-end performance. This provides a scalable and cost-efficient storage solution for future Engram-integrated LLMs without compromising inference performance.