Inter-Stance: A Dyadic Multimodal Corpus for Conversational Stance Analysis

TL;DR

This paper introduces a comprehensive multimodal dyadic interaction dataset capturing facial, vocal, physiological, and self-report data to advance social interaction analysis.

Contribution

It provides a large, annotated, multimodal corpus of social interactions with dyads having shared history or strangers, enabling new research in interpersonal behavior modeling.

Findings

Extensive experiments demonstrate differences in multimodal communication based on interpersonal history.

The dataset enables analysis of social signals, agreement, disagreement, and neutral stance.

The corpus includes 20TB of multimodal data for research use.

Abstract

Social interactions dominate our perceptions of the world and shape our daily behavior by attaching social meaning to acts as simple and spontaneous as gestures, facial expressions, voice, and speech. People mimic and otherwise respond to each other's postures, facial expressions, mannerisms, and other verbal and nonverbal behavior, and form appraisals or evaluations in the process. Yet, no publicly-available dataset includes multimodal recordings and self-report measures of multiple persons in social interaction. Dyadic recordings and annotation are lacking. We present a new data corpus of multimodal dyadic interaction (45 dyads, 90 persons) that includes synchronized multi-modality behavior (2D face video, 3D face geometry, thermal spectrum dynamics, voice and speech behavior, physiology (PPG, EDA, heart-rate, blood pressure, and respiration), and self-reported affect of all participants in a communicative interaction scenario. Two types of dyads are included: persons with shared past history and strangers. Annotations include social signals, agreement, disagreement, and neutral stance. With a potent emotion induction, these multimodal data will enable novel modeling of multimodal interpersonal behavior. We present extensive experiments to evaluate multimodal dyadic communication of dyads with and without interpersonal history, and their affect. This new database will make multimodal modeling of social interaction never possible before. The dataset includes 20TB of multimodal data to share with the research community.