Exploring Speech Foundation Models for Speaker Diarization Across Lifespan

TL;DR

This paper evaluates the robustness of speech foundation models for speaker diarization across different age groups, highlighting challenges and improvements through multi-age training and adaptation.

Contribution

It introduces a comprehensive cross-lifespan evaluation of speaker diarization models and demonstrates effective strategies for enhancing age-invariant performance.

Findings

Models trained on adult speech perform poorly on children and older adults.

Joint multi-age training improves robustness across age groups.

Targeted age group adaptation further enhances diarization accuracy, especially with Whisper encoder.

Abstract

Speech foundation models have shown strong transferability across a wide range of speech applications. However, their robustness to age-related domain shift in speaker diarization remains underexplored. In this work, we present a cross-lifespan evaluation within a unified end-to-end neural diarization framework (EEND-VC), covering speech samples from conversations involving children, adults, and older adults. We compare models under zero-shot cross-age inference, joint multi-age training, and domain-specific adaptation. Results show substantial performance degradation when models trained on adult-specific speech are applied to child and older-adult conversational data. Moreover, joint multi-age training across different age groups improves robustness without reducing diarization performance in canonical adult conversations, while targeted age group adaptation yields further gains in diarization performance, particularly when using the Whisper encoder.