CodeSep: Low-Bitrate Codec-Driven Speech Separation with Base-Token Disentanglement and Auxiliary-Token Serial Prediction

TL;DR

CodeSep introduces a novel speech separation and compression model that disentangles multiple speakers into discrete tokens, enabling low-bitrate transmission with competitive separation quality, suitable for online communication and archiving.

Contribution

It proposes a joint speech separation and low-bitrate compression framework using a residual vector quantizer and token disentanglement, which is a new approach in speech processing.

Findings

Achieves satisfactory separation at 1 kbps bitrate.

Outperforms baseline methods in separation quality.

Efficient token-based representation for speech compression.

Abstract

This paper targets a new scenario that integrates speech separation with speech compression, aiming to disentangle multiple speakers while producing discrete representations for efficient transmission or storage, with applications in online meetings and dialogue archiving. To address this scenario, we propose CodeSep, a codec-driven model that jointly performs speech separation and low-bitrate compression. CodeSep comprises a residual vector quantizer (RVQ)-based plain neural speech codec, a base-token disentanglement (BTD) module, and parallel auxiliary-token serial prediction (ATSP) modules. The BTD module disentangles mixed-speech mel-spectrograms into base tokens for each speaker, which are then refined by ATSP modules to serially predict auxiliary tokens, and finally, all tokens are decoded to reconstruct separated waveforms through the codec decoder. During training, the codec's RVQ provides supervision with permutation-invariant and teacher-forcing-based cross-entropy losses. As only base tokens are transmitted or stored, CodeSep achieves low-bitrate compression. Experimental results show that CodeSep attains satisfactory separation performance at only 1 kbps compared with baseline methods.