Speaker-conditioned Target Speaker Extraction based on Customized LSTM Cells

TL;DR

This paper introduces a novel approach for target speaker extraction using customized LSTM cells within a CNN-LSTM network, significantly enhancing extraction accuracy by focusing on target-specific voice patterns.

Contribution

It proposes a new method of customizing LSTM cells to better remember target speaker characteristics, improving extraction performance over standard LSTM models.

Findings

Customized LSTM cells outperform standard LSTM in speaker extraction tasks.

Significant performance improvement on Librispeech two-speaker mixtures.

Effective focus on target speaker voice patterns enhances extraction accuracy.

Abstract

Speaker-conditioned target speaker extraction systems rely on auxiliary information about the target speaker to extract the target speaker signal from a mixture of multiple speakers. Typically, a deep neural network is applied to isolate the relevant target speaker characteristics. In this paper, we focus on a single-channel target speaker extraction system based on a CNN-LSTM separator network and a speaker embedder network requiring reference speech of the target speaker. In the LSTM layer of the separator network, we propose to customize the LSTM cells in order to only remember the specific voice patterns corresponding to the target speaker by modifying the information processing in the forget gate. Experimental results for two-speaker mixtures using the Librispeech dataset show that this customization significantly improves the target speaker extraction performance compared to using standard LSTM cells.