Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization

TL;DR

This paper presents a method to compress self-supervised speech models for speaker diarization by fine-tuning before structured pruning, achieving significant size reduction and speedup without performance loss.

Contribution

It introduces a novel approach of fine-tuning SSL models prior to structured pruning, enabling high compression ratios with maintained accuracy.

Findings

Up to 80% parameter reduction without performance loss.

Inference speed increased by 4.0x and 2.6x on GPU.

Effective on multiple datasets including AMI, AISHELL-4, and AliMeeting.

Abstract

Self-supervised learning (SSL) models like WavLM can be effectively utilized when building speaker diarization systems but are often large and slow, limiting their use in resource constrained scenarios. Previous studies have explored compression techniques, but usually for the price of degraded performance at high pruning ratios. In this work, we propose to compress SSL models through structured pruning by introducing knowledge distillation. Different from the existing works, we emphasize the importance of fine-tuning SSL models before pruning. Experiments on far-field single-channel AMI, AISHELL-4, and AliMeeting datasets show that our method can remove redundant parameters of WavLM Base+ and WavLM Large by up to 80% without any performance degradation. After pruning, the inference speeds on a single GPU for the Base+ and Large models are 4.0 and 2.6 times faster, respectively. Our source code is publicly available.