ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

TL;DR

This paper introduces ReactMotion, a new task and dataset for generating natural listener motions in response to speaker utterances, addressing the challenge of modeling non-deterministic human reactions.

Contribution

It presents ReactMotionNet dataset capturing multiple listener responses and a unified generative framework ReactMotion trained with preference-based objectives.

Findings

ReactMotion outperforms retrieval baselines.

Generated motions are more natural and diverse.

Preference-oriented evaluation protocols improve assessment.

Abstract

In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.