DiffProxy: Multi-View Human Mesh Recovery via Diffusion-Generated Dense Proxies

TL;DR

DiffProxy leverages diffusion models to generate dense surface correspondences for multi-view human mesh recovery, achieving state-of-the-art results on real-world benchmarks by improving intermediate representations.

Contribution

It introduces a diffusion-based dense proxy generator trained on synthetic data, enhancing multi-view human mesh recovery with precise surface constraints and uncertainty estimation.

Findings

Achieves state-of-the-art performance on five benchmarks.

Uses diffusion models for dense correspondence prediction.

Provides fine-grained hand and body surface details.

Abstract

Precise human mesh recovery (HMR) from multi-view images remains challenging: end-to-end methods produce entangled errors hard to localize, while fitting-based methods rely on sparse keypoints that provide limited surface constraints. We observe that the true bottleneck lies in the quality of intermediate representations, and that dense pixel-to-surface correspondences can be effectively generated by repurposing pre-trained diffusion models with rich visual priors. We propose DiffProxy, a Stable-Diffusion-based framework trained on large-scale synthetic data with pixel-perfect annotations. A multi-conditional proxy generator predicts dense correspondences from multi-view images, providing uniform surface constraints that enable precise fitting. Hand refinement feeds enlarged hand crops alongside full-body images for fine-grained detail, while test-time scaling exploits diffusion stochasticity to estimate per-pixel uncertainty. Trained only on synthetic data, DiffProxy achieves state-of-the-art results on five diverse real-world benchmarks. Project page: https://wrk226.github.io/DiffProxy.html