MuSteerNet: Human Reaction Generation from Videos via Observation-Reaction Mutual Steering

TL;DR

MuSteerNet is a framework that improves video-driven 3D human reaction generation by addressing relational distortions between visual inputs and reaction types, using mutual steering mechanisms.

Contribution

The paper introduces MuSteerNet, which employs Prototype Feedback Steering and Dual-Coupled Reaction Refinement to enhance reaction quality from videos.

Findings

Achieves competitive performance in human reaction generation.

Effectively mitigates relational distortion between observations and reactions.

Demonstrates through experiments that the method improves reaction realism.

Abstract

Video-driven human reaction generation aims to synthesize 3D human motions that directly react to observed video sequences, which is crucial for building human-like interactive AI systems. However, existing methods often fail to effectively leverage video inputs to steer human reaction synthesis, resulting in reaction motions that are mismatched with the content of video sequences. We reveal that this limitation arises from a severe relational distortion between visual observations and reaction types. In light of this, we propose MuSteerNet, a simple yet effective framework that generates 3D human reactions from videos via observation-reaction mutual steering. Specifically, we first propose a Prototype Feedback Steering mechanism to mitigate relational distortion by refining visual observations with a gated delta-rectification modulator and a relational margin constraint, guided by prototypical vectors learned from human reactions. We then introduce Dual-Coupled Reaction Refinement that fully leverages rectified visual cues to further steer the refinement of generated reaction motions, thereby effectively improving reaction quality and enabling MuSteerNet to achieve competitive performance. Extensive experiments and ablation studies validate the effectiveness of our method. Code coming soon: https://github.com/zhouyuan888888/MuSteerNet.