Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representations

TL;DR

This paper addresses the challenge of position dependence in self-supervised speech representations during speech enhancement fine-tuning, proposing position-invariant methods that improve convergence and performance.

Contribution

It introduces position-invariant fine-tuning strategies for SSL-based speech enhancement, notably using soft-DTW loss and zero-padding, to mitigate positional bias.

Findings

Soft-DTW-based fine-tuning converges faster.

Position-invariant methods improve downstream speech enhancement performance.

Zero-padding alone is less effective than soft-DTW in this context.

Abstract

Integrating front-end speech enhancement (SE) models with self-supervised learning (SSL)-based speech models is effective for downstream tasks in noisy conditions. SE models are commonly fine-tuned using SSL representations with mean squared error (MSE) loss between enhanced and clean speech. However, MSE is prone to exploiting positional embeddings in SSL models, allowing the objective to be minimised through positional correlations instead of content-related information. This work frames the problem as a general limitation of self-supervised representation fine-tuning and investigates it through representation-guided SE. Two strategies are considered: (1) zero-padding, previously explored in SSL pre-training but here examined in the fine-tuning setting, and (2) speed perturbations with a soft-DTW loss. Experiments show that the soft-DTW-based approach achieves faster convergence and improved downstream performance, underscoring the importance of position-invariant fine-tuning in SSL-based speech modelling.