Mixture of Diverse Size Experts

TL;DR

This paper introduces MoDSE, a novel mixture-of-experts architecture with experts of varying sizes, improving language model performance and efficiency by better matching expert size to token needs.

Contribution

The paper proposes a new MoE design with diverse expert sizes and an expert-pair allocation strategy to balance workload and enhance model performance.

Findings

Experts of different sizes improve prediction accuracy.

Stable routing paths after training enhance model reliability.

Workload balancing strategy ensures efficient GPU utilization.

Abstract

The Sparsely-Activated Mixture-of-Experts (MoE) has gained increasing popularity for scaling up large language models (LLMs) without exploding computational costs. Despite its success, the current design faces a challenge where all experts have the same size, limiting the ability of tokens to choose the experts with the most appropriate size for generating the next token. In this paper, we propose the Mixture of Diverse Size Experts (MoDSE), a new MoE architecture with layers designed to have experts of different sizes. Our analysis of difficult token generation tasks shows that experts of various sizes achieve better predictions, and the routing path of the experts tends to be stable after a training period. However, having experts of diverse sizes can lead to uneven workload distribution. To tackle this limitation, we introduce an expert-pair allocation strategy to evenly distribute the workload across multiple GPUs. Comprehensive evaluations across multiple benchmarks demonstrate the effectiveness of MoDSE, as it outperforms existing MoEs by allocating the parameter budget to experts adaptively while maintaining the same total parameter size and the number of experts.