Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM

TL;DR

Yuan3.0 Ultra is a large, open-source MoE language model optimized for enterprise tasks, featuring a novel Layer-Adaptive Expert Pruning algorithm that improves training efficiency and reduces model size while maintaining high performance.

Contribution

The paper introduces Yuan3.0 Ultra, a trillion-parameter MoE LLM designed for enterprise applications, and proposes LAEP, a new expert pruning method that enhances pre-training efficiency and model compactness.

Findings

LAEP reduces model size by 33.3%

Pre-training efficiency improves by 49% with LAEP

Yuan3.0 Ultra achieves state-of-the-art results on enterprise benchmarks

Abstract

We introduce Yuan3.0 Ultra, an open-source Mixture-of-Experts (MoE) large language model featuring 68.8B activated parameters and 1010B total parameters, specially designed to enhance performance on enterprise scenarios tasks while maintaining competitive capabilities on general purpose tasks. We propose Layer-Adaptive Expert Pruning (LAEP) algorithm designed for the pre-training stage of MoE LLMs. In contrast to previous expert pruning approaches that operate primarily in the post-training phase, the proposed algorithm enhances training efficiency by selectively pruning underutilized experts and reorganizing experts across computing devices according to token distribution statistics. Comprehensive experiments demonstrate that LAEP effectively reduces model size and substantially improves pre-training efficiency. When pre-training Yuan3.0 Ultra from scratch original with 1515B parameters, this algorithm delivers a 49\% boost in pre-training efficiency and a 33.3\% reduction in total parameters, while preserving the model's outstanding multi-domain performance. On enterprise scenario benchmarks including Docmatix, ChatRAG, SummEval and MMTab, Yuan3.0 Ultra achieves leading accuracy. The model and codes are publicly available at https://github.com/Yuan-lab-LLM/Yuan3.0-Ultra.