A generic diffusion-based approach for 3D human pose prediction in the wild

TL;DR

This paper introduces a diffusion-based method for 3D human pose prediction that effectively handles noisy inputs and occlusions, outperforming existing methods on multiple datasets and enhancing other models through pre- and post-processing.

Contribution

The paper presents a novel diffusion-based framework for 3D human pose prediction that manages noisy data and occlusions, with a temporal cascaded model for long-term forecasting.

Findings

Outperforms state-of-the-art on four datasets

Handles noisy inputs and occlusions effectively

Improves existing models as a pre- or post-processing step

Abstract

Predicting 3D human poses in real-world scenarios, also known as human pose forecasting, is inevitably subject to noisy inputs arising from inaccurate 3D pose estimations and occlusions. To address these challenges, we propose a diffusion-based approach that can predict given noisy observations. We frame the prediction task as a denoising problem, where both observation and prediction are considered as a single sequence containing missing elements (whether in the observation or prediction horizon). All missing elements are treated as noise and denoised with our conditional diffusion model. To better handle long-term forecasting horizon, we present a temporal cascaded diffusion model. We demonstrate the benefits of our approach on four publicly available datasets (Human3.6M, HumanEva-I, AMASS, and 3DPW), outperforming the state-of-the-art. Additionally, we show that our framework is generic enough to improve any 3D pose prediction model as a pre-processing step to repair their inputs and a post-processing step to refine their outputs. The code is available online: \url{https://github.com/vita-epfl/DePOSit}.