Adapting Multi-Lingual ASR Models for Handling Multiple Talkers

TL;DR

This paper presents a method to adapt large-scale multilingual speech models for multi-talker ASR, enabling recognition of overlapped speech with timestamp prediction, while maintaining multilingual capabilities even with limited language data.

Contribution

The authors develop an enhanced serialized output training method combined with a lightweight adapter to adapt USMs for multi-talker recognition with timestamp prediction, preserving multilinguality.

Findings

Effective transfer of USMs to multi-talker ASR with timestamps

Maintains multilingual performance with limited adaptation data

Improves recognition accuracy on meeting conversation datasets

Abstract

State-of-the-art large-scale universal speech models (USMs) show a decent automatic speech recognition (ASR) performance across multiple domains and languages. However, it remains a challenge for these models to recognize overlapped speech, which is often seen in meeting conversations. We propose an approach to adapt USMs for multi-talker ASR. We first develop an enhanced version of serialized output training to jointly perform multi-talker ASR and utterance timestamp prediction. That is, we predict the ASR hypotheses for all speakers, count the speakers, and estimate the utterance timestamps at the same time. We further introduce a lightweight adapter module to maintain the multilingual property of the USMs even when we perform the adaptation with only a single language. Experimental results obtained using the AMI and AliMeeting corpora show that our proposed approach effectively transfers the USMs to a strong multilingual multi-talker ASR model with timestamp prediction capability.