SnapPose3D: Diffusion-Based Single-Frame 2D-to-3D Lifting of Human Poses

TL;DR

SnapPose3D introduces a diffusion-based framework for single-frame 2D-to-3D human pose lifting, generating multiple hypotheses to address depth ambiguity and joint uncertainty, achieving state-of-the-art results.

Contribution

It is the first to leverage diffusion models for multi-hypothesis 2D-to-3D pose estimation from single frames, avoiding temporal processing.

Findings

Achieves state-of-the-art results on benchmark datasets.

Effectively generates multiple plausible 3D pose hypotheses.

Improves accuracy by aggregating multiple pose predictions.

Abstract

Depth ambiguity and joint uncertainty are the two main obstacles in obtaining accurate human pose predictions by 2D-to-3D lifting methods proposed in the literature. In particular, these issues are caused by 2D joint locations that can be mapped to multiple 3D positions, inducing multiple possible final poses. Following these considerations, we propose leveraging diffusion-based models generation capability to predict multiple hypotheses and aggregate them in a final accurate pose. Therefore, we introduce SnapPose3D, a pose-lifting framework trained deterministically to denoise 3D poses conditioned on both visual context and 2D pose features. SnapPose3D adopts a probabilistic approach during inference, generating multiple hypotheses through random sampling from a unit Gaussian distribution. Unlike most previous methods that address pose ambiguity by processing temporal sequences, SnapPose3D uses single frames as input, avoiding tracking and limiting computational cost, data acquisition complexity, and the need for online, real-time applications. We extensively evaluate SnapPose3D on well-known benchmarks for the 3D human pose estimation task showing its ability to generate and aggregate accurate hypotheses that lead to state-of-the-art results.