Training Report of TeleChat3-MoE

TL;DR

This paper details the training infrastructure, optimization techniques, and parallelism strategies enabling efficient scaling of TeleChat3-MoE large language models to over one trillion parameters on Ascend NPU clusters.

Contribution

It introduces a comprehensive training infrastructure with advanced optimization and parallelism methods for large-scale MoE language models, ensuring reliable scaling and high efficiency.

Findings

Achieved near-linear scaling on thousands of devices.

Implemented advanced operator and data scheduling techniques.

Optimized multi-dimensional parallelism configurations.

Abstract

TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.