Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement

TL;DR

This paper discusses developing an AI system capable of interpreting nonverbal cues in group settings to better understand collaboration and engagement, enhancing support for social dynamics in multi-party environments.

Contribution

It introduces methods for detecting and tracking nonverbal behaviors in classroom interactions to infer common ground and engagement levels.

Findings

Effective nonverbal behavior detection in classroom settings

Implications for understanding group engagement and common ground

Potential for improving AI support in social and collaborative tasks

Abstract

Our goal is to develop an AI Partner that can provide support for group problem solving and social dynamics. In multi-party working group environments, multimodal analytics is crucial for identifying non-verbal interactions of group members. In conjunction with their verbal participation, this creates an holistic understanding of collaboration and engagement that provides necessary context for the AI Partner. In this demo, we illustrate our present capabilities at detecting and tracking nonverbal behavior in student task-oriented interactions in the classroom, and the implications for tracking common ground and engagement.