ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data

TL;DR

ReMoGen is a modular framework that enables real-time, high-fidelity human reaction generation in interactive scenarios by leveraging large-scale motion data and adaptive modules for diverse, data-scarce environments.

Contribution

It introduces ReMoGen, a novel modular learning approach that combines a universal motion prior with meta-interaction modules for robust, real-time human reaction synthesis.

Findings

ReMoGen produces high-quality, coherent reactions in diverse interaction settings.

The framework generalizes effectively across different interaction scenarios.

ReMoGen achieves low-latency responses suitable for online interaction.

Abstract

Human behaviors in real-world environments are inherently interactive, with an individual's motion shaped by surrounding agents and the scene. Such capabilities are essential for applications in virtual avatars, interactive animation, and human-robot collaboration. We target real-time human interaction-to-reaction generation, which generates the ego's future motion from dynamic multi-source cues, including others' actions, scene geometry, and optional high-level semantic inputs. This task is fundamentally challenging due to (i) limited and fragmented interaction data distributed across heterogeneous single-person, human-human, and human-scene domains, and (ii) the need to produce low-latency yet high-fidelity motion responses during continuous online interaction. To address these challenges, we propose ReMoGen (Reaction Motion Generation), a modular learning framework for real-time interaction-to-reaction generation. ReMoGen leverages a universal motion prior learned from large-scale single-person motion datasets and adapts it to target interaction domains through independently trained Meta-Interaction modules, enabling robust generalization under data-scarce and heterogeneous supervision. To support responsive online interaction, ReMoGen performs segment-level generation together with a lightweight Frame-wise Segment Refinement module that incorporates newly observed cues at the frame level, improving both responsiveness and temporal coherence without expensive full-sequence inference. Extensive experiments across human-human, human-scene, and mixed-modality interaction settings show that ReMoGen produces high-quality, coherent, and responsive reactions, while generalizing effectively across diverse interaction scenarios.