Autonomy-of-Experts Models

TL;DR

The paper introduces Autonomy-of-Experts, a new MoE paradigm where experts self-select based on activation norms, removing routers and improving expert selection and learning efficiency in language models.

Contribution

It proposes a novel MoE approach where experts autonomously select themselves, enhancing expert utilization and learning without relying on a router.

Findings

AoE outperforms traditional MoE models in language tasks.

Pre-trained models with 700M to 4B parameters show improved performance.

The approach reduces routing overhead through low-rank activation pre-computation.

Abstract

Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue that the separation between the router's decision-making and the experts' execution is a critical yet overlooked issue, leading to suboptimal expert selection and ineffective learning. To address this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which experts autonomously select themselves to process inputs. AoE is based on the insight that an expert is aware of its own capacity to effectively process a token, an awareness reflected in the scale of its internal activations. In AoE, routers are removed; instead, experts pre-compute internal activations for inputs and are ranked based on their activation norms. Only the top-ranking experts proceed with the forward pass, while the others abort. The overhead of pre-computing activations is reduced through a low-rank weight factorization. This self-evaluating-then-partner-comparing approach ensures improved expert selection and effective learning. We pre-train language models having 700M up to 4B parameters, demonstrating that AoE outperforms traditional MoE models with comparable efficiency.