Privileged Contrastive Pretraining for Multimodal Affect Modelling

TL;DR

This paper introduces PriCon, a novel affective computing framework that combines privileged contrastive pretraining with LUPI to improve model robustness and transferability from controlled to real-world environments.

Contribution

PriCon is the first framework to integrate supervised contrastive learning with privileged information in affective modeling, enhancing real-world applicability.

Findings

PriCon outperforms LUPI and end-to-end models on benchmark datasets.

PriCon achieves performance comparable to full modality models during testing.

The framework effectively bridges the gap between laboratory and real-world affective modeling.

Abstract

Affective Computing (AC) has made significant progress with the advent of deep learning, yet a persistent challenge remains: the reliable transfer of affective models from controlled laboratory settings (in-vitro) to uncontrolled real-world environments (in-vivo). To address this challenge we introduce the Privileged Contrastive Pretraining (PriCon) framework according to which models are first pretrained via supervised contrastive learning (SCL) and then act as teacher models within a Learning Using Privileged Information (LUPI) framework. PriCon both leverages privileged information during training and enhances the robustness of derived affect models via SCL. Experiments conducted on two benchmark affective corpora, RECOLA and AGAIN, demonstrate that models trained using PriCon consistently outperform LUPI and end to end models. Remarkably, in many cases, PriCon models achieve performance comparable to models trained with access to all modalities during both training and testing. The findings underscore the potential of PriCon as a paradigm towards further bridging the gap between in-vitro and in-vivo affective modelling, offering a scalable and practical solution for real-world applications.