MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

TL;DR

This paper introduces MoEfication, a method to convert Transformer feed-forward layers into Mixtures of Experts, reducing computation while preserving performance and revealing functional partitions within FFNs.

Contribution

It proposes a novel approach to transform FFNs into MoE structures, enabling efficient inference and deeper understanding of FFN mechanisms.

Findings

MoEfication activates only 10-30% of FFN parameters per input.

Maintains over 95% of original model performance after conversion.

Achieves 2x inference speedup with 25% of FFN parameters.

Abstract

Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain. To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models on various downstream tasks. Besides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs. The source code of this paper can be obtained from https://github.com/thunlp/MoEfication.