Nonverbal Interaction Detection
Jianan Wei, Tianfei Zhou, Yi Yang, Wenguan Wang

TL;DR
This paper introduces a new large-scale dataset and a hypergraph-based model for detecting and understanding complex human nonverbal interactions in images, advancing the analysis of multifaceted social cues.
Contribution
It presents the first systematic approach to nonverbal interaction detection, including a novel dataset, a new task formulation, and a hypergraph model that captures high-order social interactions.
Findings
NVI-DEHR outperforms baseline methods on NVI-DET.
The model generalizes well to related tasks like HOI-DET.
The dataset provides detailed annotations for nonverbal behaviors.
Abstract
This work addresses a new challenge of understanding human nonverbal interaction in social contexts. Nonverbal signals pervade virtually every communicative act. Our gestures, facial expressions, postures, gaze, even physical appearance all convey messages, without anything being said. Despite their critical role in social life, nonverbal signals receive very limited attention as compared to the linguistic counterparts, and existing solutions typically examine nonverbal cues in isolation. Our study marks the first systematic effort to enhance the interpretation of multifaceted nonverbal signals. First, we contribute a novel large-scale dataset, called NVI, which is meticulously annotated to include bounding boxes for humans and corresponding social groups, along with 22 atomic-level nonverbal behaviors under five broad interaction types. Second, we establish a new task NVI-DET for…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAnomaly Detection Techniques and Applications · Speech and dialogue systems · Emotion and Mood Recognition
MethodsSoftmax · Attention Is All You Need
