Streaming Speaker Change Detection and Gender Classification for Transducer-Based Multi-Talker Speech Translation

TL;DR

This paper presents a streaming multi-talker speech translation system that simultaneously detects speaker changes and classifies gender using speaker embeddings within a transducer-based model, enabling improved speaker-aware translation and synthesis.

Contribution

It introduces a novel integration of speaker embeddings into a transducer-based streaming speech translation model for joint speaker change detection and gender classification.

Findings

High accuracy achieved for speaker change detection.

Effective gender classification in streaming conditions.

Enhanced multi-talker speech translation performance.

Abstract

Streaming multi-talker speech translation is a task that involves not only generating accurate and fluent translations with low latency but also recognizing when a speaker change occurs and what the speaker's gender is. Speaker change information can be used to create audio prompts for a zero-shot text-to-speech system, and gender can help to select speaker profiles in a conventional text-to-speech model. We propose to tackle streaming speaker change detection and gender classification by incorporating speaker embeddings into a transducer-based streaming end-to-end speech translation model. Our experiments demonstrate that the proposed methods can achieve high accuracy for both speaker change detection and gender classification.