JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency

TL;DR

JoyAI-LLM Flash is a token-efficient, sparsely activated Mixture-of-Experts language model with advanced training and optimization techniques, designed for high performance in the sub-50B parameter range.

Contribution

The paper introduces JoyAI-LLM Flash, a novel MoE model with a new RL algorithm FiberPO, optimized sparsity, and joint training-inference methods for improved efficiency and performance.

Findings

Achieves high sparsity with only 2.7B active parameters out of 48B total.

Pretrained on 20 trillion tokens with advanced fine-tuning and RL techniques.

Provides open-source checkpoints on Hugging Face.

Abstract

We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B parameter regime. JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline, including supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and large-scale reinforcement learning (RL) across diverse environments. To improve token efficiency, JoyAI-LLM Flash strategically balances \emph{thinking} and \emph{non-thinking} cognitive modes and introduces FiberPO, a novel RL algorithm inspired by fibration theory that decomposes trust-region maintenance into global and local components, providing unified multi-scale stability control for LLM policy optimization. To enhance architectural sparsity, the model comprises 48B total parameters while activating only 2.7B parameters per forward pass, achieving a substantially higher sparsity ratio than contemporary industry leading models of comparable scale. To further improve inference throughput, we adopt a joint training-inference co-design that incorporates dense Multi-Token Prediction (MTP) and Quantization-Aware Training (QAT). We release the checkpoints for both JoyAI-LLM-48B-A3B Base and its post-trained variants on Hugging Face to support the open-source community.