Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

TL;DR

This paper introduces SIMoE, a novel instruction-tuning method that converts dense LLMs into sparse Mixture-of-Experts models, automatically discovering specialized experts and improving downstream task performance.

Contribution

The paper proposes SIMoE, an end-to-end algorithm for automatic expert discovery and input-dependent expert routing in LLMs, enhancing efficiency and generalization.

Findings

Achieves state-of-the-art results on instruction-tuning benchmarks.

Maintains optimal performance-compute trade-off.

Automatically identifies domain-specific experts within LLMs.

Abstract

We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.