DiffSurf: A Transformer-based Diffusion Model for Generating and Reconstructing 3D Surfaces in Pose

TL;DR

DiffSurf is a transformer-based diffusion model that effectively generates and reconstructs diverse 3D surfaces, including humans and objects, and performs well on various 3D tasks with high quality and efficiency.

Contribution

The paper introduces DiffSurf, a novel transformer-based diffusion architecture for 3D surface generation and reconstruction, enhancing diversity, quality, and versatility over prior models.

Findings

Outperforms previous models in shape diversity and quality

Achieves near real-time accuracy in 3D human mesh recovery

Versatile in handling multiple 3D tasks like morphing and shape variation

Abstract

This paper presents DiffSurf, a transformer-based denoising diffusion model for generating and reconstructing 3D surfaces. Specifically, we design a diffusion transformer architecture that predicts noise from noisy 3D surface vertices and normals. With this architecture, DiffSurf is able to generate 3D surfaces in various poses and shapes, such as human bodies, hands, animals and man-made objects. Further, DiffSurf is versatile in that it can address various 3D downstream tasks including morphing, body shape variation and 3D human mesh fitting to 2D keypoints. Experimental results on 3D human model benchmarks demonstrate that DiffSurf can generate shapes with greater diversity and higher quality than previous generative models. Furthermore, when applied to the task of single-image 3D human mesh recovery, DiffSurf achieves accuracy comparable to prior techniques at a near real-time rate.