CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing

TL;DR

CA-SSLR introduces a condition-aware self-supervised speech representation that dynamically incorporates language and speaker context, enhancing generalization and performance across diverse speech tasks with minimal tuning.

Contribution

It presents a novel condition-aware SSL model that integrates language and speaker embeddings early, reducing reliance on input features and improving adaptability to unseen tasks.

Findings

10% reduction in language identification errors

37% improvement in speech recognition CER

27% decrease in speaker verification EER

Abstract

We introduce Condition-Aware Self-Supervised Learning Representation (CA-SSLR), a generalist conditioning model broadly applicable to various speech-processing tasks. Compared to standard fine-tuning methods that optimize for downstream models, CA-SSLR integrates language and speaker embeddings from earlier layers, making the SSL model aware of the current language and speaker context. This approach reduces the reliance on input audio features while preserving the integrity of the base SSLR. CA-SSLR improves the model's capabilities and demonstrates its generality on unseen tasks with minimal task-specific tuning. Our method employs linear modulation to dynamically adjust internal representations, enabling fine-grained adaptability without significantly altering the original model behavior. Experiments show that CA-SSLR reduces the number of trainable parameters, mitigates overfitting, and excels in under-resourced and unseen tasks. Specifically, CA-SSLR achieves a 10% relative reduction in LID errors, a 37% improvement in ASR CER on the ML-SUPERB benchmark, and a 27% decrease in SV EER on VoxCeleb-1, demonstrating its effectiveness.