Multimodal Quantitative Measures for Multiparty Behaviour Evaluation

TL;DR

This paper introduces a comprehensive framework for objectively evaluating multiparty social behavior in digital humans using three novel metrics that analyze synchrony, timing, and structural similarity in skeletal motion data.

Contribution

It presents a unified, intervention-driven evaluation framework with three complementary metrics, validated through perturbation experiments and perception studies, advancing the assessment of social behavior in digital agents.

Findings

Metrics detect changes due to gesture dampening, delays, and pitch flattening.

Metrics reveal predictable shifts in social coordination measures.

The framework provides orthogonal insights into spatial, temporal, and behavioral aspects.

Abstract

Digital humans are emerging as autonomous agents in multiparty interactions, yet existing evaluation metrics largely ignore contextual coordination dynamics. We introduce a unified, intervention-driven framework for objective assessment of multiparty social behaviour in skeletal motion data, spanning three complementary dimensions: (1) synchrony via Cross-Recurrence Quantification Analysis, (2) temporal alignment via Multiscale Empirical Mode Decompositionbased Beat Consistency, and (3) structural similarity via Soft Dynamic Time Warping. We validate metric sensitivity through three theory-driven perturbations -- gesture kinematic dampening, uniform speech-gesture delays, and prosodic pitch-variance reduction-applied to $\approx 145$ 30-second thin slices of group interactions from the DnD dataset. Mixed-effects analyses reveal predictable, joint-independent shifts: dampening increases CRQA determinism and reduces beat consistency, delays weaken cross-participant coupling, and pitch flattening elevates F0 Soft-DTW costs. A complementary perception study ($N=27$) compares judgments of full-video and skeleton-only renderings to quantify representation effects. Our three measures deliver orthogonal insights into spatial structure, timing alignment, and behavioural variability. Thereby forming a robust toolkit for evaluating and refining socially intelligent agents. Code available on \href{https://github.com/tapri-lab/gig-interveners}{GitHub}.