Analysing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study

TL;DR

This study uses computer vision and causal inference to analyze the direction of emotional influence in dyadic conversations through facial expressions, addressing transient effects and subtle expressions.

Contribution

It introduces a novel method combining interval selection and Granger causality to determine emotional influence direction from facial expressions in dyadic dialogue.

Findings

Effective identification of causal influence direction over transient intervals

Facial expressions can reveal subtle emotional influence patterns

Approach performs well across different interaction conditions

Abstract

Identifying the direction of emotional influence in a dyadic dialogue is of increasing interest in the psychological sciences with applications in psychotherapy, analysis of political interactions, or interpersonal conflict behavior. Facial expressions are widely described as being automatic and thus hard to overtly influence. As such, they are a perfect measure for a better understanding of unintentional behavior cues about social-emotional cognitive processes. With this view, this study is concerned with the analysis of the direction of emotional influence in dyadic dialogue based on facial expressions only. We exploit computer vision capabilities along with causal inference theory for quantitative verification of hypotheses on the direction of emotional influence, i.e., causal effect relationships, in dyadic dialogues. We address two main issues. First, in a dyadic dialogue, emotional influence occurs over transient time intervals and with intensity and direction that are variant over time. To this end, we propose a relevant interval selection approach that we use prior to causal inference to identify those transient intervals where causal inference should be applied. Second, we propose to use fine-grained facial expressions that are present when strong distinct facial emotions are not visible. To specify the direction of influence, we apply the concept of Granger causality to the time series of facial expressions over selected relevant intervals. We tested our approach on newly, experimentally obtained data. Based on the quantitative verification of hypotheses on the direction of emotional influence, we were able to show that the proposed approach is most promising to reveal the causal effect pattern in various instructed interaction conditions.