Discriminative-Generative Target Speaker Extraction with Decoder-Only Language Models

TL;DR

This paper introduces a novel discriminative-generative two-stage framework for target speaker extraction that combines interference suppression with high-quality speech reconstruction, improving perceptual quality and naturalness.

Contribution

The paper proposes a new two-stage framework integrating discriminative and generative models for TSE, addressing limitations of existing systems in speech quality and controllability.

Findings

Achieves better balance among perceptual quality, intelligibility, and speaker consistency.

Outperforms purely discriminative or generative baselines on TSE and SE benchmarks.

Demonstrates effectiveness of collaboration strategies like joint fine-tuning and regularization.

Abstract

Target speaker extraction (TSE) aims to recover the speech of a desired speaker from a mixture given a short enrollment utterance, while speech enhancement (SE) focuses on improving speech quality under noisy conditions. Most existing TSE and SE systems are based on discriminative modeling and have shown strong interference suppression ability, but they often remain limited in perceptual quality and naturalness. To address this issue, we first introduce LauraTSE, a generative TSE model built on an autoregressive decoder-only language model. Although generative modeling is promising for quality enhancement, purely generative TSE may suffer from hallucination, content drift, and limited controllability in complex acoustic conditions. We therefore propose a discriminative-generative two-stage framework, where a discriminative front-end first produces target-related representations with strong interference suppression, and a generative back-end then reconstructs high-quality speech in the neural audio codec representation space. This design combines the controllability of discriminative extraction with the reconstruction capability of generative modeling. We further investigate several collaboration strategies for the two-stage framework, including front-end freezing, joint fine-tuning, SI-SDR regularization, and autoregressive/non-autoregressive inference. Experimental results on both TSE and SE benchmarks show that the proposed framework achieves a better balance among perceptual quality, intelligibility, and speaker consistency than purely discriminative or purely generative baselines.