Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models

TL;DR

This paper introduces a structured pruning method guided by knowledge distillation to compress self-supervised speech models for speaker diarization, achieving significant efficiency gains while maintaining or improving accuracy across multiple datasets.

Contribution

The study systematically explores model compression for SSL-based diarization, demonstrating that simple structured pruning with distillation effectively reduces model size and inference time without performance loss.

Findings

Achieves up to 80% model size reduction

Enables 4x faster inference without accuracy loss

Maintains or improves performance across diverse datasets

Abstract

Self-supervised learning (SSL) models such as WavLM have substantially advanced speaker diarization by providing rich contextual speech representations. However, the high computational and memory costs of these models hinder deployment in real-time and resource-constrained scenarios. This work presents a systematic study on compressing SSL-based diarization models through structured pruning guided by knowledge distillation. We investigate pruning objectives that target both model parameters and computational complexity, and analyze alternative strategies, showing that a simple overall pruning approach provides the best balance between efficiency and accuracy. Our method achieves up to 80% model size reduction and 4x faster inference without performance degradation. Comprehensive experiments across eight public diarization datasets demonstrate that the pruned models consistently match or surpass the performance of their uncompressed counterparts. Furthermore, we show strong out-of-domain generalization on the CHiME-6 dataset, achieving accuracy comparable to the top systems in the CHiME-7 challenge without any domain adaptation. These results highlight that structured pruning, when guided by distillation, can yield efficient and generalizable diarization systems suitable for real-world applications.