Expert Routing for Communication-Efficient MoE via Finite Expert Banks

TL;DR

This paper introduces a finite-bank approach to analyze and optimize resource-efficient sparse Mixture-of-Experts models by quantifying routing information and generalization using information-theoretic measures.

Contribution

It develops a practical framework using finite expert banks and information theory to analyze and improve expert routing efficiency in sparse MoE architectures.

Findings

Mutual information I(S;W) tracks the generalization gap.

The Xu-Raginsky bound is looser than empirical estimates.

The framework enables analysis of resource-aware MoE inference systems.

Abstract

Resource-efficient machine learning increasingly uses sparse Mixture-of-Experts (MoE) architectures, where the gate acts as both a learning component and a routing interface controlling computation, communication, and accuracy. Motivated by finite-rate interpretations of MoE gating, we treat the gate as a stochastic channel and use $I(X;T)$ to quantify the routing information available to the selected expert. To make the associated information quantities tractable beyond synthetic examples, we develop a finite-bank MNIST construction using pretrained CNN experts and a discrete, data-dependent selection rule. Since the selected model belongs to a finite candidate set, the algorithmic mutual information $I(S;W)$ admits a closed-form discrete-entropy estimator from the empirical posterior $q(W|S)$. Sweeping a data-dependence parameter $\alpha$, we observe that $\widehat I(S;W)$ monotonically tracks the generalization gap, while the Xu-Raginsky bound exhibits the expected looseness. We also compare with a uniform union-bound baseline and introduce an empirical estimator of $I(X;T)$ together with a Blahut-Arimoto procedure for tracing an accuracy-rate curve over the expert bank. The proposed framework provides a practical tool for analyzing resource-aware MoE inference systems and for interpreting $I(X;T)$ and $D(R_g)$ as design proxies for efficient expert routing.