NF3DM: Combining Neural Fields and Deformation Models for 3D Non-Rigid Motion Reconstruction

TL;DR

This paper presents NF3DM, a data-driven method that combines neural fields and deformation models to reconstruct detailed, temporally coherent 3D non-rigid motion from partial, unstructured observations, especially for near-isometric shapes.

Contribution

It introduces a novel approach that integrates implicit neural shape representations with explicit deformation models for high-fidelity, temporally consistent 3D motion reconstruction without parametric shape models.

Findings

Outperforms state-of-the-art methods on human and animal motion sequences.

Effectively reconstructs shapes from monocular depth videos.

Maintains geometric details and temporal coherence in non-rigid motion.

Abstract

We introduce a novel, data-driven approach for reconstructing temporally coherent 3D motion from unstructured and potentially partial observations of non-rigidly deforming shapes. Our goal is to achieve high-fidelity motion reconstructions for shapes that undergo near-isometric deformations, such as humans wearing loose clothing. The key novelty of our work lies in its ability to combine implicit shape representations with explicit mesh-based deformation models, enabling detailed and temporally coherent motion reconstructions without relying on parametric shape models or decoupling shape and motion. Each frame is represented as a neural field decoded from a feature space where observations over time are fused, hence preserving geometric details present in the input data. Temporal coherence is enforced with a near-isometric deformation constraint between adjacent frames that applies to the underlying surface in the neural field. Our method outperforms state-of-the-art approaches, as demonstrated by its application to human and animal motion sequences reconstructed from monocular depth videos.