Improving Speaker Diarization using Semantic Information: Joint Pairwise Constraints Propagation

TL;DR

This paper introduces a novel speaker diarization method that leverages semantic information extracted from speech content using language models, improving clustering accuracy over traditional acoustic-only approaches.

Contribution

The work proposes a new framework that integrates semantic cues via spoken language understanding modules into speaker diarization, which is a significant advancement over existing acoustic-based methods.

Findings

Consistent performance improvement over acoustic-only systems.

Effective use of semantic constraints enhances clustering accuracy.

Demonstrated on public datasets with superior results.

Abstract

Speaker diarization has gained considerable attention within speech processing research community. Mainstream speaker diarization rely primarily on speakers' voice characteristics extracted from acoustic signals and often overlook the potential of semantic information. Considering the fact that speech signals can efficiently convey the content of a speech, it is of our interest to fully exploit these semantic cues utilizing language models. In this work we propose a novel approach to effectively leverage semantic information in clustering-based speaker diarization systems. Firstly, we introduce spoken language understanding modules to extract speaker-related semantic information and utilize these information to construct pairwise constraints. Secondly, we present a novel framework to integrate these constraints into the speaker diarization pipeline, enhancing the performance of the entire system. Extensive experiments conducted on the public dataset demonstrate the consistent superiority of our proposed approach over acoustic-only speaker diarization systems.