ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

TL;DR

ExpertWeaver is a novel method that leverages GLU activation patterns in dense LLMs to efficiently convert them into high-performing sparse MoEs without additional training.

Contribution

It introduces a training-free framework that uncovers inherent MoE structures in dense models using GLU activation patterns, improving conversion quality.

Findings

Outperforms existing dense-to-MoE conversion methods.

Effectively identifies universal and specialized neurons for MoE construction.

Enhances both dynamic pruning and downcycling strategies.

Abstract

Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: \textbf{dynamic structural pruning} that converts dense models into MoE architectures with moderate sparsity to balance performance and inference efficiency, and \textbf{downcycling} approaches that use pretrained dense models to initialize highly sparse MoE architectures. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) mechanism provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neural-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Our experiments demonstrate that ExpertWeaver significantly outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for superior MoE initialization.