ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling

TL;DR

ZipMoE is a novel on-device MoE serving system that uses lossless compression and cache-aware scheduling to significantly reduce latency and increase throughput on resource-limited edge devices.

Contribution

It introduces a cache-scheduling co-design that leverages hardware and parameter redundancy to enable efficient, lossless MoE inference on edge devices.

Findings

Achieves up to 72.77% latency reduction

Realizes up to 6.76× higher throughput

Demonstrates effectiveness on real-world workloads

Abstract

While Mixture-of-Experts (MoE) architectures substantially bolster the expressive power of large-language models, their prohibitive memory footprint severely impedes the practical deployment on resource-constrained edge devices, especially when model behavior must be preserved without relying on lossy quantization. In this paper, we present ZipMoE, an efficient and semantically lossless on-device MoE serving system. ZipMoE exploits the synergy between the hardware properties of edge devices and the statistical redundancy inherent to MoE parameters via a caching-scheduling co-design with provable performance guarantee. Fundamentally, our design shifts the paradigm of on-device MoE inference from an I/O-bound bottleneck to a compute-centric workflow that enables efficient parallelization. We implement a prototype of ZipMoE and conduct extensive experiments on representative edge computing platforms using popular open-source MoE models and real-world workloads. Our evaluation reveals that ZipMoE achieves up to $72.77\%$ inference latency reduction and up to $6.76\times$ higher throughput than the state-of-the-art systems.