VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

TL;DR

VisMMoE introduces a system that leverages visual-expert affinity through token pruning to enhance the efficiency of visual-language MoE model deployment on memory-limited platforms, achieving significant speedups.

Contribution

The paper proposes a novel token pruning technique based on visual-expert affinity to improve expert locality and prefetching in VL-MoE models, enabling more efficient inference.

Findings

VisMMoE achieves up to 2.68x inference speedup.

Token pruning reduces expert working set size and stabilizes expert access patterns.

The system maintains competitive accuracy while improving efficiency.

Abstract

Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as \textit{visual-expert affinity}: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.