Mixture Compressor for Mixture-of-Experts LLMs Gains More

TL;DR

This paper introduces MC, a training-free compression method for MoE-LLMs that significantly reduces memory and computation by adaptive quantization and dynamic token expert selection, with minimal accuracy loss.

Contribution

The paper proposes a novel, training-free Mixture-Compressor (MC) that combines adaptive quantization and dynamic pruning to compress MoE-LLMs efficiently.

Findings

76.6% model compression at 2.54 bits with 3.8% accuracy loss

15% reduction in activated parameters during inference with less than 0.6% performance drop

Effective trade-off between model efficiency and accuracy demonstrated through extensive experiments

Abstract

Mixture-of-Experts large language models (MoE-LLMs) marks a significant step forward of language models, however, they encounter two critical challenges in practice: 1) expert parameters lead to considerable memory consumption and loading latency; and 2) the current activated experts are redundant, as many tokens may only require a single expert. Motivated by these issues, we investigate the MoE-LLMs and make two key observations: a) different experts exhibit varying behaviors on activation reconstruction error, routing scores, and activated frequencies, highlighting their differing importance, and b) not all tokens are equally important -- only a small subset is critical. Building on these insights, we propose MC, a training-free Mixture-Compressor for MoE-LLMs, which leverages the significance of both experts and tokens to achieve an extreme compression. First, to mitigate storage and loading overheads, we introduce Pre-Loading Mixed-Precision Quantization, which formulates the adaptive bit-width allocation as a Linear Programming problem, where the objective function balances multi-factors reflecting the importance of each expert. Additionally, we develop Online Dynamic Pruning, which identifies important tokens to retain and dynamically select activated experts for other tokens during inference to optimize efficiency while maintaining performance. Our MC integrates static quantization and dynamic pruning to collaboratively achieve extreme compression for MoE-LLMs with less accuracy loss, ensuring an optimal trade-off between performance and efficiency. Extensive experiments confirm the effectiveness of our approach. For instance, at 2.54 bits, MC compresses 76.6% of the model, with only a 3.8% average accuracy loss. During dynamic inference, we further reduce activated parameters by 15%, with a performance drop of less than 0.6%.