Loose coupling of spectral and spatial models for multi-channel diarization and enhancement of meetings in dynamic environments

TL;DR

This paper introduces a novel joint spatial and spectral mixture model for multi-channel diarization and enhancement, effectively handling speaker movement in dynamic environments by loosely coupling the two models.

Contribution

It proposes a new probabilistic framework that loosely couples spectral and spatial models, enabling improved diarization and enhancement in dynamic meeting scenarios.

Findings

Significant improvements over tightly coupled models on LibriCSS data.

Effective handling of speaker position changes during meetings.

Enhanced performance in multi-channel meeting transcription.

Abstract

Sound capture by microphone arrays opens the possibility to exploit spatial, in addition to spectral, information for diarization and signal enhancement, two important tasks in meeting transcription. However, there is no one-to-one mapping of positions in space to speakers if speakers move. Here, we address this by proposing a novel joint spatial and spectral mixture model, whose two submodels are loosely coupled by modeling the relationship between speaker and position index probabilistically. Thus, spatial and spectral information can be jointly exploited, while at the same time allowing for speakers speaking from different positions. Experiments on the LibriCSS data set with simulated speaker position changes show great improvements over tightly coupled subsystems.