Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation

TL;DR

Ming-UniAudio introduces a unified speech model and tokenizer that effectively combines understanding, generation, and editing capabilities, achieving state-of-the-art results and enabling natural language-guided speech editing.

Contribution

The paper presents MingTok-Audio, the first continuous speech tokenizer integrating semantic and acoustic features, and Ming-UniAudio, a unified model for speech understanding, generation, and editing, along with a new benchmark for speech editing.

Findings

Achieves SOTA on 8 out of 12 metrics on ContextASR.

Attains Seed-TTS-WER of 0.95 for Chinese voice cloning.

Enables universal, free-form speech editing guided by natural language.

Abstract

Existing speech models suffer from competing requirements on token representations by understanding and generation tasks. This discrepancy in representation prevents speech language models from performing instruction-based free-form editing. To solve this challenge, we introduce a novel framework that unifies speech understanding, generation, and editing. The core of our unified model is a unified continuous speech tokenizer MingTok-Audio, the first continuous tokenizer to effectively integrate semantic and acoustic features, which makes it suitable for both understanding and generation tasks. Based on this unified continuous audio tokenizer, we developed the speech language model Ming-UniAudio, which achieved a balance between generation and understanding capabilities. Ming-UniAudio sets new state-of-the-art (SOTA) records on 8 out of 12 metrics on the ContextASR benchmark. Notably, for Chinese voice cloning, it achieves a highly competitive Seed-TTS-WER of 0.95. Leveraging this foundational model, we further trained a dedicated speech editing model Ming-UniAudio-Edit, the first speech language model that enables universal, free-form speech editing guided solely by natural language instructions, handling both semantic and acoustic modifications without timestamp condition. To rigorously assess the editing capability and establish a foundation for future research, we introduce Ming-Freeform-Audio-Edit, the first comprehensive benchmark tailored for instruction-based free-form speech editing, featuring diverse scenarios and evaluation dimensions spanning semantic correctness, acoustic quality, and instruction alignment. We open-sourced the continuous audio tokenizer, the unified foundational model, and the free-form instruction-based editing model to facilitate the development of unified audio understanding, generation, and manipulation.