Dual-LoRA: Parameter-Efficient Adversarial Disentanglement for Cross-Lingual Speaker Verification

TL;DR

Dual-LoRA introduces a language-anchored adversarial approach with task-factorized adapters to improve cross-lingual speaker verification, reducing language-speaker entanglement and enhancing discriminability.

Contribution

It proposes a novel language-anchored adversarial disentanglement method using trainable adapters within a frozen backbone, addressing language-speaker entanglement issues.

Findings

Achieved 0.91% validation EER on TidyVoice benchmark.

Placed 3rd in the official challenge.

Effectively disentangled language and speaker features.

Abstract

Cross-lingual speaker verification suffers from severe language-speaker entanglement. This causes systematic degradation in the hardest scenario: correctly accepting utterances from the same speaker across different languages while rejecting those from different speakers sharing the same language. Standard adversarial disentanglement degrades speaker discriminability; blind discriminators inadvertently penalize speaker-discriminative traits that merely correlate with language. To address this, we propose Dual-LoRA, injecting trainable task-factorized LoRA adapters into a frozen pre-trained backbone. Our core innovation is a Language-Anchored Adversary: by grounding the discriminator with an explicit language branch, adversarial gradients target true linguistic cues rather than arbitrary correlations, preserving essential speaker characteristics. Evaluated on the TidyVoice benchmark, our system achieves a 0.91% validation EER and achieves 3rd place in the official challenge.