HMoE: Heterogeneous Mixture of Experts for Language Modeling

TL;DR

This paper introduces HMoE, a heterogeneous mixture of experts model with experts of different sizes, improving specialization and efficiency in language modeling by encouraging activation of smaller experts and outperforming traditional homogeneous MoE models.

Contribution

The paper proposes a novel HMoE model with experts of varying sizes and a training objective to balance expert activation, enhancing efficiency and performance.

Findings

HMoE achieves lower loss with fewer activated parameters.

HMoE outperforms homogeneous MoE models on multiple benchmarks.

The approach improves expert specialization and computational efficiency.

Abstract

Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous experts, each with identical capacity. However, varying complexity in input data necessitates experts with diverse capabilities, while homogeneous MoE hinders effective expert specialization and efficient parameter utilization. In this study, we propose a novel Heterogeneous Mixture of Experts (HMoE), where experts differ in size and thus possess diverse capacities. This heterogeneity allows for more specialized experts to handle varying token complexities more effectively. To address the imbalance in expert activation, we propose a novel training objective that encourages the frequent activation of smaller experts, enhancing computational efficiency and parameter utilization. Extensive experiments demonstrate that HMoE achieves lower loss with fewer activated parameters and outperforms conventional homogeneous MoE models on various pre-training evaluation benchmarks. Codes will be released upon acceptance.