FloE: On-the-Fly MoE Inference on Memory-constrained GPU

TL;DR

FloE introduces an on-the-fly MoE inference system that compresses expert parameters and reduces memory usage, enabling efficient deployment on memory-limited GPUs with significant speedup and minimal performance loss.

Contribution

FloE presents a novel compression-based approach for MoE inference on GPUs, addressing memory constraints and latency issues in resource-limited environments.

Findings

Achieves 9.3x parameter compression per expert.

Enables deployment on 11GB VRAM GPU, reducing memory footprint by 8.5x.

Provides 48.7x inference speedup with minimal performance degradation.

Abstract

With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices. While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as a potential solution, the large size of activated experts overburdens the limited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios. To mitigate this, we propose FloE, an on-the-fly MoE inference system on memory-constrained GPUs. FloE is built on the insight that there exists substantial untapped redundancy within sparsely activated experts. It employs various compression techniques on the expert's internal parameter matrices to reduce the data movement load, combined with low-cost sparse prediction, achieving perceptible inference acceleration in wall-clock time on resource-constrained devices. Empirically, FloE achieves a 9.3x compression of parameters per expert in Mixtral-8x7B; enables deployment on a GPU with only 11GB VRAM, reducing the memory footprint by up to 8.5x; and delivers a 48.7x inference speedup compared to DeepSpeed-MII on a single GeForce RTX 3090 - all with only a 4.4$\%$ - 7.6$\%$ average performance degradation.