ReaLB: Real-Time Load Balancing for Multimodal MoE Inference

TL;DR

ReaLB is a real-time load balancing method for multimodal MoE inference that dynamically adjusts expert computation precision to improve efficiency without extra memory or redundant experts.

Contribution

It introduces a zero-overhead, runtime precision adjustment technique for load balancing in multimodal MoE inference systems.

Findings

Achieves 1.10×-1.32× speedup in inference.

Limits accuracy degradation to within 1%.

Effectively balances expert workloads during inference.

Abstract

Mixture-of-Experts (MoE) architectures are widely used in modern large language models and multimodal models. However, inference efficiency is often limited by highly dynamic and skewed expert workloads across different modalities. During the prefill stage with large batch sizes, vision tokens frequently dominate the input sequences. Under expert parallelism (EP), this leads to severe load imbalance, where a subset of devices becomes overloaded, reducing overall system throughput. We propose ReaLB, a real-time load balancing method for multimodal MoE (MMoE) inference that introduces zero scheduling overhead. ReaLB dynamically adjusts the computation precision of MoE experts at runtime on a per-EP-rank basis. For ranks dominated by vision-heavy experts, ReaLB assigns lower-precision computation to improve execution efficiency by exploiting FP4 Tensor Cores. ReaLB does not require redundant experts or additional memory allocation. Instead, it performs layer-wise expert precision transformation on the fly and hides the associated overhead within the dispatch phase before MoE computation. Experiments on representative MMoE models show that ReaLB achieves 1.10$\times$-1.32$\times$ end-to-end speedup while limiting average accuracy degradation to within 1%.