PaCMO: Partner Dependent Human Motion Generation in Dyadic Human Activity using Neural Operators

TL;DR

This paper introduces PaCMO, a neural operator-based model that generates one actor's 3D motion in dyadic activities based on the other's motion, addressing a novel, complex problem in human motion synthesis.

Contribution

It proposes a partner conditioned motion operator (PaCMO) that learns motion relationships in function spaces and introduces the F^2ID metric for evaluating generated motion quality.

Findings

PaCMO produces realistic dyadic motion sequences.

The model outperforms baselines on NTU RGB+D and DuetDance datasets.

User studies confirm the realism of generated motions.

Abstract

We address the problem of generating 3D human motions in dyadic activities. In contrast to the concurrent works, which mainly focus on generating the motion of a single actor from the textual description, we generate the motion of one of the actors from the motion of the other participating actor in the action. This is a particularly challenging, under-explored problem, that requires learning intricate relationships between the motion of two actors participating in an action and also identifying the action from the motion of one actor. To address these, we propose partner conditioned motion operator (PaCMO), a neural operator-based generative model which learns the distribution of human motion conditioned by the partner's motion in function spaces through adversarial training. Our model can handle long unlabeled action sequences at arbitrary time resolution. We also introduce the "Functional Frechet Inception Distance" ($F^2ID$) metric for capturing similarity between real and generated data for function spaces. We test PaCMO on NTU RGB+D and DuetDance datasets and our model produces realistic results evidenced by the $F^2ID$ score and the conducted user study.