Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs

TL;DR

This paper introduces NPUMoE, a runtime engine that accelerates Mixture-of-Experts LLM inference on Apple Silicon NPUs by optimizing expert routing and execution, significantly improving latency and energy efficiency.

Contribution

NPUMoE enables efficient offloading of MoE inference to Apple Silicon NPUs, overcoming dynamic routing and irregular operator challenges with novel static and load-aware techniques.

Findings

NPUMoE reduces latency by up to 5.55x

Energy efficiency improves by up to 7.37x

CPU-cycle usage decreases by up to 5.54x

Abstract

Apple Neural Engine (ANE) is a dedicated neural processing unit (NPU) present in every Apple Silicon chip. Mixture-of-Experts (MoE) LLMs improve inference efficiency via sparse activation but are challenging for NPUs in three ways: expert routing is unpredictable and introduces dynamic tensor shapes that conflict with the shape-specific constraints of NPUs; several irregular operators, e.g., top-k, scatter/gather, etc., are not NPU-friendly; and launching many small expert kernels incurs substantial dispatch and synchronization overhead. NPUs are designed to offload AI compute from CPU and GPU; our goal is to enable such offloading for MoE inference, particularly during prefill, where long-context workloads consume substantial system resources. This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon by offloading dense, static computation to NPU, while preserving a CPU/GPU fallback path for dynamic operations. NPUMoE uses offline calibration to estimate expert capacity and popularity that drives three key techniques: (1) Static tiers for expert capacity to address dynamic expert routing (2) Grouped expert execution to mitigate NPU concurrency limits (3) Load-aware expert compute graph residency to reduce CPU-NPU synchronization overhead. Experiments on Apple M-series devices using three representative MoE LLMs and four long-context workloads show that NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.