WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

TL;DR

WavCube is a unified speech model that combines understanding, reconstruction, and generation by compressing SSL features into a semantic-acoustic latent, enabling diverse speech tasks with high efficiency.

Contribution

WavCube introduces a two-stage training scheme to create a compact, unified speech representation supporting multiple tasks, improving over existing SSL features for generative modeling.

Findings

Approaches WavLM performance on SUPERB despite 8x compression.

Achieves state-of-the-art zero-shot TTS with faster convergence.

Excels in speech enhancement, separation, and voice conversion.

Abstract

Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube.