Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation

TL;DR

This paper introduces PCMECL, a novel method that enhances speech-preserving facial expression manipulation by personalizing and aligning visual-semantic correlations using cross-modal learning, addressing data scarcity issues.

Contribution

The proposed PCMECL algorithm personalizes prompts for individual expressions and aligns visual-semantic features through differencing, improving supervision in SPFEM tasks.

Findings

PCMECL outperforms existing methods across multiple datasets.

Personalized prompts capture individual expressive variations.

Feature differencing improves modality alignment and manipulation quality.

Abstract

Speech-preserving facial expression manipulation (SPFEM) aims to enhance human expressiveness without altering mouth movements tied to the original speech. A primary challenge in this domain is the scarcity of paired data, namely aligned frames of the same individual with identical speech but different expressions, which impedes direct supervision for emotional manipulation. While current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision, their direct application is limited. To this end, we propose a Personalized Cross-Modal Emotional Correlation Learning (PCMECL) algorithm that refines VLM-based supervision through two major improvements. First, standard VLMs rely on a single generic prompt for each emotion, failing to capture expressive variations among individuals. PCMECL addresses this limitation by conditioning on individual visual information to learn personalized prompts, thereby establishing more fine-grained visual-semantic correlations. Second, even with personalization, inherent discrepancies persist between the visual and semantic feature distributions. To bridge this modality gap, PCMECL employs feature differencing to correlate the modalities, providing more precisely aligned supervision by matching the change in visual features to the change in semantic features. As a plug-and-play module, PCMECL can be seamlessly integrated into existing SPFEM models. Extensive experiments across various datasets demonstrate the superior efficacy of our algorithm.