FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning

TL;DR

FastMMoE is a training-free framework that accelerates multimodal large language models by reducing redundant visual tokens through expert activation reduction and routing-aware token pruning, significantly lowering FLOPs while maintaining high performance.

Contribution

It introduces a novel, training-free acceleration method for MoE-based MLLMs that combines expert activation reduction and routing-aware token pruning from a routing analysis perspective.

Findings

FLOPs reduced by up to 55% with minimal performance loss

Outperforms dense-model pruning baselines like FastV and SparseVLM

Maintains approximately 95.5% of original performance

Abstract

Multimodal large language models (MLLMs) have achieved impressive performance, but high-resolution visual inputs result in long sequences of visual tokens and substantial inference latency. Reducing redundant visual tokens is critical to ease computational/memory burdens while preserving performance, enabling MLLM deployment in resource-constrained or latency-sensitive scenarios. Current visual token pruning methods mainly rely on attention-based redundancy analysis and are tailored to dense architectures. We propose Fast Multimodal Mixture-of-Experts (FastMMoE), a training-free acceleration framework for mixture-of-experts (MoE) based MLLMs, developed from a routing analysis perspective. FastMMoE combines two complementary strategies: (i) expert activation reduction for visual tokens to minimize unnecessary expert computation; and (ii) routing-aware token pruning that leverages similarity in routing probability distributions to identify and remove highly redundant visual tokens. Experiments on large-scale MoE-MLLMs such as DeepSeek-VL2 and InternVL3.5 demonstrate that FastMMoE can reduce FLOPs by up to 55.0% while retaining approximately 95.5% of the original performance, consistently outperforming dense-model pruning baselines including FastV and SparseVLM across multiple retention rates.