PAFUSE: Part-based Diffusion for 3D Whole-Body Pose Estimation

TL;DR

PAFUSE introduces a hierarchical part-based diffusion method for 3D whole-body pose estimation, effectively capturing fine-grained keypoints across body parts and leveraging temporal information to outperform existing approaches.

Contribution

The paper presents a novel hierarchical part-based diffusion approach that models fine-grained keypoints and exploits temporal data for improved 3D pose estimation.

Findings

Significantly outperforms state-of-the-art on H3WB dataset.

Shows improvements over other spatiotemporal 3D human pose methods.

Effectively captures part-specific details across the whole body.

Abstract

We introduce a novel approach for 3D whole-body pose estimation, addressing the challenge of scale -- and deformability -- variance across body parts brought by the challenge of extending the 17 major joints on the human body to fine-grained keypoints on the face and hands. In addition to addressing the challenge of exploiting motion in unevenly sampled data, we combine stable diffusion to a hierarchical part representation which predicts the relative locations of fine-grained keypoints within each part (e.g., face) with respect to the part's local reference frame. On the H3WB dataset, our method greatly outperforms the current state of the art, which fails to exploit the temporal information. We also show considerable improvements compared to other spatiotemporal 3D human-pose estimation approaches that fail to account for the body part specificities. Code is available at https://github.com/valeoai/PAFUSE.