Equilibria: Fair Multi-Tenant CXL Memory Tiering At Scale

TL;DR

Equilibria is an OS framework that enables fair, multi-tenant CXL memory tiering at datacenter scale, improving performance and fairness for hyperscaler workloads.

Contribution

It introduces a novel OS-level solution with per-container controls, observability, and fairness policies for multi-tenant CXL memory tiering, addressing limitations of existing solutions.

Findings

Improves performance over Linux TPP by up to 52% for production workloads.

Achieves 1.7x performance improvement for benchmarks.

Enables fair memory allocation and observability at datacenter scale.

Abstract

Memory dominates datacenter system cost and power. Memory expansion via Compute Express Link (CXL) is an effective way to provide additional memory at lower cost and power, but its effective use requires software-level tiering for hyperscaler workloads. Existing tiering solutions, including current Linux support, face fundamental limitations in production deployments. First, they lack multi-tenancy support, failing to handle stacked homogeneous or heterogeneous workloads. Second, limited control-plane flexibility leads to fairness violations and performance variability. Finally, insufficient observability prevents operators from diagnosing performance pathologies at scale. We present Equilibria, an OS framework enabling fair, multi-tenant CXL tiering at datacenter scale. Equilibria provides per-container controls for memory fair-share allocation and fine-grained observability of tiered-memory usage and operations. It further enforces flexible, user-specified fairness policies through regulated promotion and demotion, and mitigates noisy-neighbor interference by suppressing thrashing. Evaluated in a large hyperscaler fleet using production workloads and benchmarks, Equilibria helps workloads meet service level objectives (SLOs) while avoiding performance interference. It improves performance over the state-of-the-art Linux solution, TPP, by up to 52% for production workloads and 1.7x for benchmarks. All Equilibria patches have been released to the Linux community.