HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference

TL;DR

HAP introduces a dynamic hybrid parallelism approach for MoE inference, optimizing performance across various models and hardware by adaptively selecting parallel strategies using ILP, leading to significant speedups.

Contribution

This work presents HAP, a novel adaptive parallelism method that hierarchically decomposes MoE models and employs ILP to optimize inference efficiency across diverse scenarios.

Findings

Achieves up to 1.77x speedup over TP strategy on GPUs.

Maintains high performance across different MoE models.

Demonstrates effective generalization to various hardware and model configurations.

Abstract

Current inference systems for Mixture-of-Experts (MoE) models primarily employ static parallelization strategies. However, these static approaches cannot consistently achieve optimal performance across different inference scenarios, as they lack the flexibility to adapt to varying computational requirements. In this work, we propose HAP (Hybrid Adaptive Parallelism), a novel method that dynamically selects hybrid parallel strategies to enhance MoE inference efficiency. The fundamental innovation of HAP lies in hierarchically decomposing MoE architectures into two distinct computational modules: the Attention module and the Expert module, each augmented with a specialized inference latency simulation model. This decomposition promotes the construction of a comprehensive search space for seeking model parallel strategies. By leveraging Integer Linear Programming (ILP), HAP could solve the optimal hybrid parallel configurations to maximize inference efficiency under varying computational constraints. Our experiments demonstrate that HAP consistently determines parallel configurations that achieve comparable or superior performance to the TP strategy prevalent in mainstream inference systems. Compared to the TP-based inference, HAP-based inference achieves speedups of 1.68x, 1.77x, and 1.57x on A100, A6000, and V100 GPU platforms, respectively. Furthermore, HAP showcases remarkable generalization capability, maintaining performance effectiveness across diverse MoE model configurations, including Mixtral and Qwen series models.