ButterflyViT: 354$\times$ Expert Compression for Edge Vision Transformers

TL;DR

ButterflyViT introduces a novel expert compression method for Vision Transformers that significantly reduces memory requirements by geometrically reorienting shared parameters, enabling edge deployment with minimal accuracy loss.

Contribution

The paper proposes ButterflyViT, a geometric expert compression technique that achieves sub-linear memory scaling for Vision Transformers, addressing the linear memory bottleneck on edge devices.

Findings

Achieves 354× memory reduction on CIFAR-100 with 64 experts.

Maintains negligible accuracy loss despite significant compression.

Enables multiple experts to operate on memory-constrained edge devices.

Abstract

Deploying sparse Mixture of Experts(MoE) Vision Transformers remains a challenge due to linear expert memory scaling. Linear memory scaling stores $N$ independent expert weight matrices requiring $\mathcal{O}(N_E \cdot d^2)$ memory, which exceeds edge devices memory budget. Current compression methods like quantization, pruning and low-rank factorization reduce constant factors but leave the scaling bottleneck unresolved. We introduce ButterflyViT, a method that treats experts not as independent weight matrices but as geometric reorientations of a unified shared quantized substrate. Diversity among experts arises from viewing different angles of shared capacity, not from redundant storage. By applying learned rotations to a shared ternary prototype, each expert yields $\mathcal{O}(d_{\text{model}} \cdot d_{\text{ff}} + N_E \cdot n_\ell \cdot d)$ memory which is sub-linear in the number of experts. To address the unique challenges of vision, a spatial smoothness regulariser is introduced that penalises routing irregularities between adjacent patch tokens, turning patch correlation into a training signal. Across image classification tasks on CIFAR-100, ButterflyViT achieves 354$\times$ memory reduction at 64 experts with negligible accuracy loss. ButterflyViT allows multiple experts to fit on edge-constrained devices showing that geometric parameterization breaks linear scaling.