Joint ASR and Speaker Role Tagging with Serialized Output Training

TL;DR

This paper introduces a method using serialized output training to enable a single model to perform both speech recognition and speaker role tagging simultaneously, improving accuracy in conversational AI tasks.

Contribution

The paper proposes augmenting Whisper with role-specific tokens and fine-tuning it with serialized output training for joint ASR and speaker role tagging.

Findings

Over 10% reduction in multi-talker WER

Effective role-aware transcriptions in a single decoding pass

Outperforms previous self-supervised baseline

Abstract

Automatic Speech Recognition systems have made significant progress with large-scale pre-trained models. However, most current systems focus solely on transcribing the speech without identifying speaker roles, a function that is critical for conversational AI. In this work, we investigate the use of serialized output training (SOT) for joint ASR and speaker role tagging. By augmenting Whisper with role-specific tokens and fine-tuning it with SOT, we enable the model to generate role-aware transcriptions in a single decoding pass. We compare the SOT approach against a self-supervised previous baseline method on two real-world conversational datasets. Our findings show that this approach achieves more than 10% reduction in multi-talker WER, demonstrating its feasibility as a unified model for speaker-role aware speech transcription.