Towards Intercultural Affect Recognition: Audio-Visual Affect Recognition in the Wild Across Six Cultures

TL;DR

This study investigates the effectiveness of intercultural affect recognition models using audio-visual data from six cultures, finding that intercultural models can perform as well or better than intracultural ones, especially with visual cues.

Contribution

It is the first systematic study of intercultural affect recognition, introducing an attention-based feature selection method and demonstrating the potential of intercultural models without annotated data.

Findings

Intercultural models perform as well or better than intracultural models.

Visual facial features are more useful than audio cues for affect recognition.

Intercultural affect recognition is feasible without culture-specific annotated datasets.

Abstract

In our multicultural world, affect-aware AI systems that support humans need the ability to perceive affect across variations in emotion expression patterns across cultures. These systems must perform well in cultural contexts without annotated affect datasets available for training models. A standard assumption in affective computing is that affect recognition models trained and used within the same culture (intracultural) will perform better than models trained on one culture and used on different cultures (intercultural). We test this assumption and present the first systematic study of intercultural affect recognition models using videos of real-world dyadic interactions from six cultures. We develop an attention-based feature selection approach under temporal causal discovery to identify behavioral cues that can be leveraged in intercultural affect recognition models. Across all six cultures, our findings demonstrate that intercultural affect recognition models were as effective or more effective than intracultural models. We identify and contribute useful behavioral features for intercultural affect recognition; facial features from the visual modality were more useful than the audio modality in this study's context. Our paper presents a proof-of-concept and motivation for the future development of intercultural affect recognition systems, especially those deployed in low-resource situations without annotated data.