Privileged Information for Modeling Affect In The Wild

TL;DR

This paper introduces a privileged information framework that enables affect models trained with multiple modalities in laboratory settings to operate effectively in real-world scenarios using only limited data, such as pixels.

Contribution

The paper proposes a novel privileged information approach allowing affect models trained with multiple modalities to perform well using only available data in the wild.

Findings

Models trained with all modalities achieve similar accuracy when tested with limited modalities.

Privileged information enables effective affect modeling in real-world settings without intrusive sensors.

The approach advances affect interaction in natural, uncontrolled environments.

Abstract

A key challenge of affective computing research is discovering ways to reliably transfer affect models that are built in the laboratory to real world settings, namely in the wild. The existing gap between in vitro and in vivo affect applications is mainly caused by limitations related to affect sensing including intrusiveness, hardware malfunctions, availability of sensors, but also privacy and security. As a response to these limitations in this paper we are inspired by recent advances in machine learning and introduce the concept of privileged information for operating affect models in the wild. The presence of privileged information enables affect models to be trained across multiple modalities available in a lab setting and ignore modalities that are not available in the wild with no significant drop in their modeling performance. The proposed privileged information framework is tested in a game arousal corpus that contains physiological signals in the form of heart rate and electrodermal activity, game telemetry, and pixels of footage from two dissimilar games that are annotated with arousal traces. By training our arousal models using all modalities (in vitro) and using solely pixels for testing the models (in vivo), we reach levels of accuracy obtained from models that fuse all modalities both for training and testing. The findings of this paper make a decisive step towards realizing affect interaction in the wild.