SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference

TL;DR

SliceMoE introduces a novel energy-efficient MoE inference framework that employs bit-sliced expert caching and mixed-precision quantization to enable efficient on-device deployment with reduced energy and latency.

Contribution

The paper proposes SliceMoE, which combines dynamic bit-sliced caching, a new quantization scheme, and predictive cache warmup for efficient MoE inference under miss-rate constraints.

Findings

Reduces decode-stage energy consumption by up to 2.85x.

Improves decode latency by up to 1.81x.

Maintains near-high-bit accuracy with energy-efficient caching.

Abstract

MoE models offer efficient scaling through conditional computation, but their large parameter size and expensive expert offloading make on-device deployment challenging. Existing acceleration techniques such as prefetching or expert clustering often increase energy usage or reduce expert diversity. We present SliceMoE, an energy-efficient MoE inference framework for miss-rate-constrained deployment. SliceMoE introduces Dynamic Bit-Sliced Caching (DBSC), which caches experts at slice-level granularity and assigns precision on demand to expand effective expert capacity. To support mixed-precision experts without memory duplication, we propose Calibration-Free Asymmetric Matryoshka Quantization (AMAT), a truncation-based scheme that maintains compatibility between low-bit and high-bit slices. We further introduce Predictive Cache Warmup (PCW) to reduce early-decode cold misses by reshaping cache contents during prefill. Evaluated on DeepSeek-V2-Lite and Qwen1.5-MoE-A2.7B, SliceMoE reduces decode-stage energy consumption by up to 2.37x and 2.85x, respectively, and improves decode latency by up to 1.81x and 1.64x, while preserving near-high-bit accuracy. These results demonstrate that slice-level caching enables an efficient on-device MoE deployment.