SpecMD: A Comprehensive Study On Speculative Expert Prefetching

TL;DR

This paper introduces SpecMD, a benchmarking framework for expert cache policies in Mixture-of-Experts models, revealing that traditional policies like LRU are ineffective and proposing a new policy that significantly improves cache performance.

Contribution

The paper develops SpecMD, a standardized benchmarking framework, and proposes Least-Stale, a novel cache eviction policy tailored for MoE models, demonstrating substantial performance improvements.

Findings

MoE expert access does not follow temporal locality assumptions.

Least-Stale reduces collision misses by up to 85× compared to LRU.

Achieves over 88% hit rate with significant TTFT reduction.

Abstract

Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model's parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop \textbf{SpecMD}, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose \textbf{Least-Stale}, a novel eviction policy that exploits MoE's predictable expert access patterns to reduce collision misses by up to $85\times$ over LRU. With such gains, we achieve over $88\%$ hit rates with up to $34.7\%$ Time-to-first-token (TTFT) reduction on OLMoE at only $5\%$ or $0.6GB$ of VRAM cache capacity.