Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation

TL;DR

This paper introduces BBSI, a new annotated dataset of complex bodily behaviors in social interactions, and evaluates state-of-the-art models for automatic detection, highlighting challenges and future directions.

Contribution

The paper presents the first annotated dataset of complex bodily behaviors in social interactions and adapts advanced models for behavior detection in continuous group settings.

Findings

Promising detection results with room for improvement

Comprehensive dataset with 26 hours of annotated social behavior

Evaluation of multiple spatial-temporal feature variants

Abstract

Body language is an eye-catching social signal and its automatic analysis can significantly advance artificial intelligence systems to understand and actively participate in social interactions. While computer vision has made impressive progress in low-level tasks like head and body pose estimation, the detection of more subtle behaviors such as gesturing, grooming, or fumbling is not well explored. In this paper we present BBSI, the first set of annotations of complex Bodily Behaviors embedded in continuous Social Interactions in a group setting. Based on previous work in psychology, we manually annotated 26 hours of spontaneous human behavior in the MPIIGroupInteraction dataset with 15 distinct body language classes. We present comprehensive descriptive statistics on the resulting dataset as well as results of annotation quality evaluations. For automatic detection of these behaviors, we adapt the Pyramid Dilated Attention Network (PDAN), a state-of-the-art approach for human action detection. We perform experiments using four variants of spatial-temporal features as input to PDAN: Two-Stream Inflated 3D CNN, Temporal Segment Networks, Temporal Shift Module and Swin Transformer. Results are promising and indicate a great room for improvement in this difficult task. Representing a key piece in the puzzle towards automatic understanding of social behavior, BBSI is fully available to the research community.