Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems

TL;DR

This paper presents a context-aware, CXL-enabled GPU-NDP system for Mixture-of-Experts models that significantly improves inference throughput by optimizing expert placement and quantization, reducing memory transfer costs.

Contribution

It introduces a novel context-aware MoE inference system with dynamic expert placement and mixed-precision quantization on CXL-NDP, enhancing throughput and efficiency.

Findings

Achieves up to 8.7x decoding throughput improvement

Only 0.13% average accuracy drop

Effectively overlaps GPU and NDP execution

Abstract

Mixture-of-Experts (MoE) models scale large language models through conditional computation, but inference becomes memory-bound once expert weights exceed the capacity of GPU memory. In this case, weights must be offloaded to external memory, and fetching them incurs costly and repeated transfers. We address this by adopting CXL-attached near-data processing (CXL-NDP) as the offloading tier to execute cold experts in place, converting expensive parameter movement into cheaper activation movement. Unlike prior GPU-NDP systems that are largely context-agnostic and reactive, we develop a context-aware MoE system that uses prefill-stage activation statistics to guide decoding-stage expert placement, dynamically pins hot experts in GPU-side HBM, and maps the remainder to CXL-NDP. To meet NDP's limited compute throughput, we introduce context-aware mixed-precision quantization that allocates per-expert bitwidths (1-4 bit) based on prefill stage. The resulting MoE inference system overlaps GPU and NDP execution while minimizing cross-device movement. The evaluation on the GPU-NDP system shows that our approach achieves up to an 8.7-fold decoding throughput improvement over the state-of-the-art method, while incurring only a 0.13% average accuracy drop.