LongCat-Flash-Omni Technical Report

TL;DR

LongCat-Flash-Omni is a large-scale, open-source omni-modal model with 560 billion parameters, capable of real-time audio-visual interaction through a novel curriculum-inspired training strategy and efficient multimodal modules.

Contribution

It introduces a scalable, efficient training scheme and architecture for a 560B parameter omni-modal model, advancing open-source multimodal AI capabilities.

Findings

Achieves state-of-the-art performance on omni-modal benchmarks.

Maintains over 90% of text-only training throughput.

Delivers competitive results across diverse modality-specific tasks.

Abstract

We introduce LongCat-Flash-Omni, a state-of-the-art open-source omni-modal model with 560 billion parameters, excelling at real-time audio-visual interaction. By adopting a curriculum-inspired progressive training strategy that transitions from simpler to increasingly complex modality sequence modeling tasks, LongCat-Flash-Omni attains comprehensive multimodal capabilities while maintaining strong unimodal capability. Building upon LongCat-Flash, which adopts a high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture with zero-computation experts, LongCat-Flash-Omni integrates efficient multimodal perception and speech reconstruction modules. Despite its immense size of 560B parameters (with 27B activated), LongCat-Flash-Omni achieves low-latency real-time audio-visual interaction. For training infrastructure, we developed a modality-decoupled parallelism scheme specifically designed to manage the data and model heterogeneity inherent in large-scale multimodal training. This innovative approach demonstrates exceptional efficiency by sustaining over 90% of the throughput achieved by text-only training. Extensive evaluations show that LongCat-Flash-Omni achieves state-of-the-art performance on omni-modal benchmarks among open-source models. Furthermore, it delivers highly competitive results across a wide range of modality-specific tasks, including text, image, and video understanding, as well as audio understanding and generation. We provide a comprehensive overview of the model architecture design, training procedures, and data strategies, and open-source the model to foster future research and development in the community.