MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models

TL;DR

This paper presents a novel method for multilingual speaker attribution in automatic speech recognition that uses a frozen ASR model and speaker embeddings trained on monolingual data, achieving competitive results.

Contribution

It introduces a speaker attribution approach leveraging a frozen multilingual ASR model and weakly labeled speaker embeddings without modifying the ASR system.

Findings

Effective speaker attribution across multilingual datasets.

Competitive performance with existing methods.

Robustness to overlapping speech scenarios.

Abstract

Speaker-attributed automatic speech recognition (SA-ASR) aims to transcribe speech while assigning transcripts to the corresponding speakers accurately. Existing methods often rely on complex modular systems or require extensive fine-tuning of joint modules, limiting their adaptability and general efficiency. This paper introduces a novel approach, leveraging a frozen multilingual ASR model to incorporate speaker attribution into the transcriptions, using only standard monolingual ASR datasets. Our method involves training a speaker module to predict speaker embeddings based on weak labels without requiring additional ASR model modifications. Despite being trained exclusively with non-overlapping monolingual data, our approach effectively extracts speaker attributes across diverse multilingual datasets, including those with overlapping speech. Experimental results demonstrate competitive performance compared to strong baselines, highlighting the model's robustness and potential for practical applications.