Direct Reward Fine-Tuning on Poses for Single Image to 3D Human in the Wild

TL;DR

This paper introduces DrPose, a novel fine-tuning method that improves 3D human pose reconstruction from single images by leveraging a new pose-focused dataset and a differentiable reward to enhance pose accuracy without needing extensive 3D data.

Contribution

The paper presents DrPose, a post-training fine-tuning approach that uses a new pose dataset and a differentiable reward to improve 3D human reconstruction in challenging poses.

Findings

Significant improvement in pose accuracy on benchmarks.

Enhanced reconstruction quality for dynamic and challenging poses.

Effective use of pose data without requiring 3D assets.

Abstract

Single-view 3D human reconstruction has achieved remarkable progress through the adoption of multi-view diffusion models, yet the recovered 3D humans often exhibit unnatural poses. This phenomenon becomes pronounced when reconstructing 3D humans with dynamic or challenging poses, which we attribute to the limited scale of available 3D human datasets with diverse poses. To address this limitation, we introduce DrPose, Direct Reward fine-tuning algorithm on Poses, which enables post-training of a multi-view diffusion model on diverse poses without requiring expensive 3D human assets. DrPose trains a model using only human poses paired with single-view images, employing a direct reward fine-tuning to maximize PoseScore, which is our proposed differentiable reward that quantifies consistency between a generated multi-view latent image and a ground-truth human pose. This optimization is conducted on DrPose15K, a novel dataset that was constructed from an existing human motion dataset and a pose-conditioned video generative model. Constructed from abundant human pose sequence data, DrPose15K exhibits a broader pose distribution compared to existing 3D human datasets. We validate our approach through evaluation on conventional benchmark datasets, in-the-wild images, and a newly constructed benchmark, with a particular focus on assessing performance on challenging human poses. Our results demonstrate consistent qualitative and quantitative improvements across all benchmarks. Project page: https://seunguk-do.github.io/drpose.