Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR

TL;DR

This paper introduces a generative target speech extraction method using the pre-trained Whisper model, combining semantic and acoustic modeling to improve speech intelligibility and perceptual quality in overlapped multi-talker scenarios.

Contribution

It proposes a novel generative TSE framework based on Whisper, integrating semantic and flow-based acoustic modeling for enhanced speech extraction.

Findings

Outperforms existing baselines on multiple benchmarks

Achieves higher speech intelligibility and perceptual quality

Demonstrates effectiveness through speech samples and evaluations

Abstract

Target speech extraction (TSE) isolates the speech of a specific speaker from a multi-talker overlapped speech mixture. Most existing TSE models rely on discriminative methods, typically predicting a time-frequency spectrogram mask for the target speech. However, imperfections in these masks often result in over-/under-suppression of target/non-target speech, degrading perceptual quality. Generative methods, by contrast, re-synthesize target speech based on the mixture and target speaker cues, achieving superior perceptual quality. Nevertheless, these methods often overlook speech intelligibility, leading to alterations or loss of semantic content in the re-synthesized speech. Inspired by the Whisper model's success in target speaker ASR, we propose a generative TSE framework based on the pre-trained Whisper model to address the above issues. This framework integrates semantic modeling with flow-based acoustic modeling to achieve both high intelligibility and perceptual quality. Results from multiple benchmarks demonstrate that the proposed method outperforms existing generative and discriminative baselines. We present speech samples on https://aisaka0v0.github.io/GenerativeTSE_demo/.