$\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

TL;DR

This paper introduces $b$-MoE, a dynamic, test-time pruning method that adaptively reduces model complexity based on task-specific prompts, improving efficiency without retraining.

Contribution

It proposes a novel mixture of micro-experts framework enabling adaptive, prompt-dependent pruning during inference, addressing domain shift issues in activation-aware compression.

Findings

$b$-MoE achieves effective task-specific sparsity.

It reduces inference complexity dynamically.

Demonstrates adaptability across various downstream tasks.

Abstract

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these rely on calibration data, domain shift may arise for unknown downstream tasks. With a computationally efficient calibration, activation-aware pruning can be executed for every prompt adaptively, yet achieving reduced complexity at inference. We formulate it as a mixture of micro-experts, called $\mu$-MoE. Several experiments demonstrate that $\mu$-MoE can dynamically adapt to task/prompt-dependent structured sparsity on the fly.