EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast

TL;DR

EmotionRankCLAP introduces a supervised contrastive learning method that captures the ordinal nature of emotions in speech and language, improving cross-modal alignment and emotion understanding.

Contribution

It proposes a novel Rank-N-Contrast objective to model emotion orderings, addressing limitations of previous CLAP methods in capturing emotion ordinality.

Findings

Outperforms existing emotion-CLAP methods in cross-modal retrieval.

Effectively models fine-grained emotion variations.

Enhances understanding of emotional speech and language relationships.

Abstract

Current emotion-based contrastive language-audio pretraining (CLAP) methods typically learn by na\"ively aligning audio samples with corresponding text prompts. Consequently, this approach fails to capture the ordinal nature of emotions, hindering inter-emotion understanding and often resulting in a wide modality gap between the audio and text embeddings due to insufficient alignment. To handle these drawbacks, we introduce EmotionRankCLAP, a supervised contrastive learning approach that uses dimensional attributes of emotional speech and natural language prompts to jointly capture fine-grained emotion variations and improve cross-modal alignment. Our approach utilizes a Rank-N-Contrast objective to learn ordered relationships by contrasting samples based on their rankings in the valence-arousal space. EmotionRankCLAP outperforms existing emotion-CLAP methods in modeling emotion ordinality across modalities, measured via a cross-modal retrieval task.