DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts

TL;DR

This paper introduces DIVE, a novel method for reconstructing large language models into Mixture-of-Experts architectures that emphasizes expert diversity, reducing training costs while maintaining high accuracy.

Contribution

The paper proposes a diversity-aware reconstruction approach called DIVE, incorporating domain affinity, pruning, and efficient retraining to improve MoE model reconstruction from dense LLMs.

Findings

DIVE outperforms existing methods in training efficiency.

DIVE maintains high accuracy with minimal trade-offs.

DIVE effectively enhances expert diversity during reconstruction.

Abstract

Large language models (LLMs) with the Mixture-of-Experts (MoE) architecture achieve high cost-efficiency by selectively activating a subset of the parameters. Despite the inference efficiency of MoE LLMs, the training of extensive experts from scratch incurs substantial overhead, whereas reconstructing a dense LLM into an MoE LLM significantly reduces the training budget. However, existing reconstruction methods often overlook the diversity among experts, leading to potential redundancy. In this paper, we come up with the observation that a specific LLM exhibits notable diversity after being pruned on different calibration datasets, based on which we present a Diversity-Enhanced reconstruction method named DIVE. The recipe of DIVE includes domain affinity mining, pruning-based expert reconstruction, and efficient retraining. Specifically, the reconstruction includes pruning and reassembly of the feed-forward network (FFN) module. After reconstruction, we efficiently retrain the model on routers, experts and normalization modules. We implement DIVE on Llama-style LLMs with open-source training corpora. Experiments show that DIVE achieves training efficiency with minimal accuracy trade-offs, outperforming existing pruning and MoE reconstruction methods with the same number of activated parameters.