Supervised Contrastive Learning for Affect Modelling

TL;DR

This paper explores the use of supervised contrastive learning to develop general, subject-invariant affect representations that improve the accuracy of affect modeling, specifically for arousal prediction across multiple modalities.

Contribution

It introduces three supervised contrastive learning methods for affect representation and demonstrates their effectiveness in enhancing arousal prediction accuracy in a multimodal dataset.

Findings

Contrastive learning boosts affect prediction accuracy.

Representations are general-purpose and subject-agnostic.

Methods outperform end-to-end classification approaches.

Abstract

Affect modeling is viewed, traditionally, as the process of mapping measurable affect manifestations from multiple modalities of user input to affect labels. That mapping is usually inferred through end-to-end (manifestation-to-affect) machine learning processes. What if, instead, one trains general, subject-invariant representations that consider affect information and then uses such representations to model affect? In this paper we assume that affect labels form an integral part, and not just the training signal, of an affect representation and we explore how the recent paradigm of contrastive learning can be employed to discover general high-level affect-infused representations for the purpose of modeling affect. We introduce three different supervised contrastive learning approaches for training representations that consider affect information. In this initial study we test the proposed methods for arousal prediction in the RECOLA dataset based on user information from multiple modalities. Results demonstrate the representation capacity of contrastive learning and its efficiency in boosting the accuracy of affect models. Beyond their evidenced higher performance compared to end-to-end arousal classification, the resulting representations are general-purpose and subject-agnostic, as training is guided though general affect information available in any multimodal corpus.