Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR

TL;DR

This paper introduces a self-speaker adaptation method for multi-talker streaming ASR that adapts to individual speakers without explicit enrollment, improving recognition accuracy in overlapped speech scenarios.

Contribution

The paper presents a novel self-speaker adaptation technique that dynamically generates speaker-specific kernels using speaker supervision, eliminating the need for explicit speaker queries.

Findings

Achieves state-of-the-art performance in offline and streaming multi-talker ASR.

Effectively handles fully overlapped speech with instantaneous speaker adaptation.

Demonstrates robustness in severe overlapping speech conditions.

Abstract

We propose a self-speaker adaptation method for streaming multi-talker automatic speech recognition (ASR) that eliminates the need for explicit speaker queries. Unlike conventional approaches requiring target speaker embeddings or enrollment audio, our technique dynamically adapts individual ASR instances through speaker-wise speech activity prediction. The key innovation involves injecting speaker-specific kernels generated via speaker supervision activations into selected ASR encoder layers. This enables instantaneous speaker adaptation to target speakers while handling fully overlapped speech even in a streaming scenario. Experiments show state-of-the-art performance in both offline and streaming scenarios, demonstrating that our self-adaptive method effectively addresses severe speech overlap through streamlined speaker-focused recognition. The results validate the proposed self-speaker adaptation approach as a robust solution for multi-talker ASR under severe overlapping speech conditions.