SiftMoE: Similarity-Aware Energy-Efficient Expert Selection for Wireless Distributed MoE Inference

TL;DR

SiftMoE introduces a similarity-aware expert selection method for wireless distributed MoE inference, reducing energy consumption and communication costs while maintaining accuracy in resource-constrained edge networks.

Contribution

The paper proposes a novel expert selection framework based on expert similarity, with theoretical bounds and optimal policies for different channel conditions in wireless distributed MoE.

Findings

Significant energy savings compared to traditional methods.

Maintains inference accuracy despite expert skipping.

Effective in both slow-fading and fast-fading wireless channels.

Abstract

Mixture-of-Experts (MoE) architectures leverage sparse activation to enhance the scalability of large language models (LLMs), making them suitable for deployment in resource-constrained edge networks. However, the sheer number of experts often exceeds the memory capacity of individual edge nodes, necessitating wireless distributed MoE (WIDE) inference where experts are spread across multiple edge nodes. In this context, expert selection directly affects communication costs. Motivated by the similarity of experts, we propose SiftMoE, which judiciously selects or skips experts to strike a tradeoff between communication costs and inference accuracy. Specifically, we first establish theoretical bounds on the accuracy degradation resulting from expert replacement or skipping. Based on the bounds, we formulate an energy minimization problem for expert selection in WIDE inference subject to latency and accuracy constraints. In particular, for slow-fading channels, we derive optimal expert selection policies for both single-token decoding and multi-token prefilling. For fast-fading channels, we further extend our scheme to cope with rapidly varying channel conditions. Simulation results demonstrate that SiftMoE significantly reduces energy consumption while maintaining inference accuracy compared with conventional Top-K routing in WIDE systems.