MuSACo: Multimodal Subject-Specific Selection and Adaptation for Expression Recognition with Co-Training

TL;DR

MuSACo introduces a multimodal, subject-specific co-training approach for expression recognition that leverages multiple sources and modalities to improve accuracy and robustness in personalized affective computing applications.

Contribution

It proposes a novel multimodal subject-specific selection and adaptation method using co-training, explicitly capturing individual differences and utilizing multiple modalities for better expression recognition.

Findings

MuSACo outperforms existing MSDA and UDA methods on multiple datasets.

It effectively leverages multimodal information for personalized expression recognition.

The approach improves recognition accuracy in challenging real-world datasets.

Abstract

Personalized expression recognition (ER) involves adapting a machine learning model to subject-specific data for improved recognition of expressions with considerable interpersonal variability. Subject-specific ER can benefit significantly from multi-source domain adaptation (MSDA) methods, where each domain corresponds to a specific subject to improve model accuracy and robustness. Despite promising results, state-of-the-art MSDA approaches often overlook multimodal information or blend sources into a single domain, limiting subject diversity and failing to explicitly capture unique subject-specific characteristics. To address these limitations, we introduce MuSACo, a multimodal subject-specific selection and adaptation method for ER based on co-training. It leverages complementary information across multiple modalities and multiple source domains for subject-specific adaptation. This makes MuSACo particularly relevant for affective computing applications in digital health, such as patient-specific assessment for stress or pain, where subject-level nuances are crucial. MuSACo selects source subjects relevant to the target and generates pseudo-labels using the dominant modality for class-aware learning, in conjunction with a class-agnostic loss to learn from less confident target samples. Finally, source features from each modality are aligned, while only confident target features are combined. Experimental results on challenging multimodal ER datasets: BioVid, StressID, and BAH show that MuSACo outperforms UDA (blending) and state-of-the-art MSDA methods.