EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts
Anzhe Cheng, Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Shahin Nazarian, Paul Thompson, Paul Bogdan

TL;DR
EMoE introduces an eigenbasis-guided routing mechanism for Mixture-of-Experts models, promoting balanced expert utilization and diversity without auxiliary loss, thereby addressing key limitations of existing approaches.
Contribution
The paper proposes EMoE, a novel architecture using a learned eigenbasis for routing, which inherently balances load and encourages expert specialization without auxiliary losses.
Findings
Promotes balanced expert utilization
Encourages diverse, specialized experts
Eliminates need for auxiliary load-balancing loss
Abstract
The relentless scaling of deep learning models has led to unsustainable computational demands, positioning Mixture-of-Experts (MoE) architectures as a promising path towards greater efficiency. However, MoE models are plagued by two fundamental challenges: 1) a load imbalance problem known as the``rich get richer" phenomenon, where a few experts are over-utilized, and 2) an expert homogeneity problem, where experts learn redundant representations, negating their purpose. Current solutions typically employ an auxiliary load-balancing loss that, while mitigating imbalance, often exacerbates homogeneity by enforcing uniform routing at the expense of specialization. To resolve this, we introduce the Eigen-Mixture-of-Experts (EMoE), a novel architecture that leverages a routing mechanism based on a learned orthonormal eigenbasis. EMoE projects input tokens onto this shared eigenbasis and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Domain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications
