GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

TL;DR

GenTSE introduces a two-stage generative language model for target speaker extraction, improving speech quality, intelligibility, and consistency by separating semantic and acoustic modeling and employing advanced training strategies.

Contribution

The paper proposes a novel two-stage decoder-only generative LM approach for TSE that separates semantics and acoustics, and introduces training strategies to enhance decoding stability and output quality.

Findings

Outperforms previous LM-based TSE systems in speech quality.

Achieves higher intelligibility and speaker consistency.

Demonstrates effectiveness on Libri2Mix dataset.

Abstract

Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech. We present GenTSE, a two-stage decoder-only generative LM approach for TSE: Stage-1 predicts coarse semantic tokens, and Stage-2 generates fine acoustic tokens. Separating semantics and acoustics stabilizes decoding and yields more faithful, content-aligned target speech. Both stages use continuous SSL or codec embeddings, offering richer context than discretized-prompt methods. To reduce exposure bias, we employ a Frozen-LM Conditioning training strategy that conditions the LMs on predicted tokens from earlier checkpoints to reduce the gap between teacher-forcing training and autoregressive inference. We further employ DPO to better align outputs with human perceptual preferences. Experiments on Libri2Mix show that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.