DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs

TL;DR

DALI is a workload-aware offloading framework that dynamically assigns experts to CPU or GPU, improves prefetching accuracy, and enhances cache efficiency, enabling efficient MoE inference on resource-constrained local PCs.

Contribution

DALI introduces a dynamic expert assignment, residual-based prefetching, and workload-aware cache policies tailored for MoE inference on local PCs, addressing key inefficiencies in existing methods.

Findings

Significant speedups in prefill and decoding phases.

Effective expert workload prediction improves resource utilization.

Enhanced GPU cache hit rates lead to better inference performance.

Abstract

Mixture of Experts (MoE) architectures significantly enhance the capacity of LLMs without proportional increases in computation, but at the cost of a vast parameter size. Offloading MoE expert parameters to host memory and leveraging both CPU and GPU computation has recently emerged as a promising direction to support such models on resourceconstrained local PC platforms. While promising, we notice that existing approaches mismatch the dynamic nature of expert workloads, which leads to three fundamental inefficiencies: (1) Static expert assignment causes severe CPUGPU load imbalance, underutilizing CPU and GPU resources; (2) Existing prefetching techniques fail to accurately predict high-workload experts, leading to costly inaccurate prefetches; (3) GPU cache policies neglect workload dynamics, resulting in poor hit rates and limited effectiveness. To address these challenges, we propose DALI, a workloaDAware offLoadIng framework for efficient MoE inference on local PCs. To fully utilize hardware resources, DALI first dynamically assigns experts to CPU or GPU by modeling assignment as a 0-1 integer optimization problem and solving it efficiently using a Greedy Assignment strategy at runtime. To improve prefetching accuracy, we develop a Residual-Based Prefetching method leveraging inter-layer residual information to accurately predict high-workload experts. Additionally, we introduce a Workload-Aware Cache Replacement policy that exploits temporal correlation in expert activations to improve GPU cache efficiency. By evaluating across various MoE models and settings, DALI achieves significant speedups in the both prefill and decoding phases over the state-of-the-art offloading frameworks.