Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses

TL;DR

This paper introduces a novel pose representation learning method for 3D deformable objects that enables pose disentanglement, transfer, and diverse shape generation without explicit shape parameterization or correspondence supervision.

Contribution

It proposes a keypoint-based hybrid pose representation and an implicit deformation field for pose transfer and generation, improving flexibility and accuracy over prior methods.

Findings

State-of-the-art pose transfer performance

Ability to generate diverse deformed shapes

Effective pose disentanglement without explicit shape models

Abstract

We propose a novel method for learning representations of poses for 3D deformable objects, which specializes in 1) disentangling pose information from the object's identity, 2) facilitating the learning of pose variations, and 3) transferring pose information to other object identities. Based on these properties, our method enables the generation of 3D deformable objects with diversity in both identities and poses, using variations of a single object. It does not require explicit shape parameterization such as skeletons or joints, point-level or shape-level correspondence supervision, or variations of the target object for pose transfer. To achieve pose disentanglement, compactness for generative models, and transferability, we first design the pose extractor to represent the pose as a keypoint-based hybrid representation and the pose applier to learn an implicit deformation field. To better distill pose information from the object's geometry, we propose the implicit pose applier to output an intrinsic mesh property, the face Jacobian. Once the extracted pose information is transferred to the target object, the pose applier is fine-tuned in a self-supervised manner to better describe the target object's shapes with pose variations. The extracted poses are also used to train a cascaded diffusion model to enable the generation of novel poses. Our experiments with the DeformThings4D and Human datasets demonstrate state-of-the-art performance in pose transfer and the ability to generate diverse deformed shapes with various objects and poses.