DNCASR: End-to-End Training for Speaker-Attributed ASR

TL;DR

DNCASR is an end-to-end system that jointly performs speaker clustering and speech recognition for multi-party meetings, improving accuracy in overlapping speech scenarios through linked decoders and serialised training.

Contribution

It introduces a novel end-to-end trainable architecture with linked decoders for joint speaker attribution and ASR, addressing overlapping speech effectively.

Findings

Outperforms non-linked systems on AMI-MDM corpus

Achieves 9.0% relative reduction in speaker-attributed WER

Effectively handles overlapping speech in meetings

Abstract

This paper introduces DNCASR, a novel end-to-end trainable system designed for joint neural speaker clustering and automatic speech recognition (ASR), enabling speaker-attributed transcription of long multi-party meetings. DNCASR uses two separate encoders to independently encode global speaker characteristics and local waveform information, along with two linked decoders to generate speaker-attributed transcriptions. The use of linked decoders allows the entire system to be jointly trained under a unified loss function. By employing a serialised training approach, DNCASR effectively addresses overlapping speech in real-world meetings, where the link improves the prediction of speaker indices in overlapping segments. Experiments on the AMI-MDM meeting corpus demonstrate that the jointly trained DNCASR outperforms a parallel system that does not have links between the speaker and ASR decoders. Using cpWER to measure the speaker-attributed word error rate, DNCASR achieves a 9.0% relative reduction on the AMI-MDM Eval set.