NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action

TL;DR

NeMo introduces a novel method for reconstructing 3D human motion from multiple videos of the same action, leveraging shared information to outperform existing monocular methods in accuracy and robustness.

Contribution

The paper proposes the Neural Motion (NeMo) field, a new approach that improves 3D motion recovery by utilizing shared information across multiple video instances of the same action.

Findings

NeMo outperforms existing HMR methods in 2D keypoint detection.

NeMo achieves better 3D reconstruction metrics on a collected MoCap dataset.

NeMo effectively captures dynamic motions in sports videos.

Abstract

The task of reconstructing 3D human motion has wideranging applications. The gold standard Motion capture (MoCap) systems are accurate but inaccessible to the general public due to their cost, hardware and space constraints. In contrast, monocular human mesh recovery (HMR) methods are much more accessible than MoCap as they take single-view videos as inputs. Replacing the multi-view Mo- Cap systems with a monocular HMR method would break the current barriers to collecting accurate 3D motion thus making exciting applications like motion analysis and motiondriven animation accessible to the general public. However, performance of existing HMR methods degrade when the video contains challenging and dynamic motion that is not in existing MoCap datasets used for training. This reduces its appeal as dynamic motion is frequently the target in 3D motion recovery in the aforementioned applications. Our study aims to bridge the gap between monocular HMR and multi-view MoCap systems by leveraging information shared across multiple video instances of the same action. We introduce the Neural Motion (NeMo) field. It is optimized to represent the underlying 3D motions across a set of videos of the same action. Empirically, we show that NeMo can recover 3D motion in sports using videos from the Penn Action dataset, where NeMo outperforms existing HMR methods in terms of 2D keypoint detection. To further validate NeMo using 3D metrics, we collected a small MoCap dataset mimicking actions in Penn Action,and show that NeMo achieves better 3D reconstruction compared to various baselines.