Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

TL;DR

Grove MoE introduces a heterogeneous expert architecture with dynamic activation, improving efficiency and performance of large language models by activating only relevant parameters based on input complexity.

Contribution

The paper proposes Grove MoE, a novel architecture with experts of varying sizes and a dynamic activation mechanism, enhancing efficiency and scalability of MoE-based LLMs.

Findings

Grove MoE models activate 3.14-3.28B parameters based on token complexity.

Grove MoE models achieve comparable performance to larger SOTA models.

Dynamic expert activation improves computational efficiency.

Abstract

The Mixture of Experts (MoE) architecture is a cornerstone of modern state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate scalability by enabling sparse parameter activation. However, traditional MoE architecture uses homogeneous experts of a uniform size, activating a fixed number of parameters irrespective of input complexity and thus limiting computational efficiency. To overcome this limitation, we introduce Grove MoE, a novel architecture incorporating experts of varying sizes, inspired by the heterogeneous big.LITTLE CPU architecture. This architecture features novel adjugate experts with a dynamic activation mechanism, enabling model capacity expansion while maintaining manageable computational overhead. Building on this architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model during mid-training and post-training. GroveMoE models dynamically activate 3.14-3.28B parameters based on token complexity and achieve performance comparable to SOTA open-source models of similar or even larger size.