Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems

TL;DR

Sortformer introduces a permutation-resolving loss and architecture for improved speaker diarization and multi-speaker speech-to-text, enhancing accuracy and integration into larger models.

Contribution

It proposes a novel Sort Loss and a streamlined multi-speaker speech-to-text architecture that effectively address speaker permutation issues in speech-to-text systems.

Findings

Sort Loss improves speaker diarization performance.

Incorporating speaker supervision enhances multi-speaker transcription accuracy.

The approach facilitates seamless integration of speaker tagging into speech-to-text models.

Abstract

Sortformer is an encoder-based speaker diarization model designed for supervising speaker tagging in speech-to-text models. Instead of relying solely on permutation invariant loss (PIL), Sortformer introduces Sort Loss to resolve the permutation problem, either independently or in tandem with PIL. In addition, we propose a streamlined multi-speaker speech-to-text architecture that leverages Sortformer for speaker supervision, embedding speaker labels into the encoder using sinusoidal kernel functions. This design addresses the speaker permutation problem through sorted objectives, effectively bridging timestamps and tokens to supervise speaker labels in the output transcriptions. Experiments demonstrate that Sort Loss can boost speaker diarization performance, and incorporating the speaker supervision from Sortformer improves multi-speaker transcription accuracy. We anticipate that the proposed Sortformer and multi-speaker architecture will enable the seamless integration of speaker tagging capabilities into foundational speech-to-text systems and multimodal large language models (LLMs), offering an easily adoptable and user-friendly mechanism to enhance their versatility and performance in speaker-aware tasks. The code and trained models are made publicly available through the NVIDIA NeMo Framework.