ExFusion: Efficient Transformer Training via Multi-Experts Fusion

TL;DR

ExFusion is a novel pre-training method that efficiently leverages multi-expert fusion in Transformer models, enhancing performance with minimal additional computational cost and no extra deployment overhead.

Contribution

It introduces a multi-expert fusion approach during Transformer pre-training that maintains efficiency and reduces deployment complexity.

Findings

Effective multi-expert fusion improves Transformer training efficiency.

Achieves comparable performance with reduced computational overhead.

Simplifies deployment by consolidating experts post-training.

Abstract

Mixture-of-Experts (MoE) models substantially improve performance by increasing the capacity of dense architectures. However, directly training MoE models requires considerable computational resources and introduces extra overhead in parameter storage and deployment. Therefore, it is critical to develop an approach that leverages the multi-expert capability of MoE to enhance performance while incurring minimal additional cost. To this end, we propose a novel pre-training approach, termed ExFusion, which improves the efficiency of Transformer training through multi-expert fusion. Specifically, during the initialization phase, ExFusion upcycles the feed-forward network (FFN) of the Transformer into a multi-expert configuration, where each expert is assigned a weight for later parameter fusion. During training, these weights allow multiple experts to be fused into a single unified expert equivalent to the original FFN, which is subsequently used for forward computation. As a result, ExFusion introduces multi-expert characteristics into the training process while incurring only marginal computational cost compared to standard dense training. After training, the learned weights are used to integrate multi-experts into a single unified expert, thereby eliminating additional overhead in storage and deployment. Extensive experiments on a variety of computer vision and natural language processing tasks demonstrate the effectiveness of the proposed method.