Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour

TL;DR

This paper presents a novel approach to automatically detect low rapport in small group interactions using non-verbal cues, achieving promising accuracy and analyzing the influence of different features and temporal segments.

Contribution

First to investigate automatic low rapport detection in small groups using multi-modal non-verbal signals and personality information.

Findings

Facial features enable 0.7 average precision in low rapport detection

Incorporating personality knowledge allows early prediction without performance loss

Different non-verbal features and temporal segments vary in informational content

Abstract

Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants' personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.