VI3DRM:Towards meticulous 3D Reconstruction from Sparse Views via Photo-Realistic Novel View Synthesis

TL;DR

VI3DRM introduces a diffusion-based 3D reconstruction method that disentangles semantic, color, material, and lighting factors to produce highly realistic 3D models from sparse views, surpassing previous approaches.

Contribution

The paper presents VI3DRM, a novel diffusion model operating in a perspective-disentangled latent space for realistic 3D reconstruction from sparse views, improving realism and accuracy.

Findings

Outperforms DreamComposer on GSO dataset with higher PSNR, SSIM, and lower LPIPS.

Generates highly realistic, photo-like 3D images from sparse views.

Enables detailed textured meshes and point clouds.

Abstract

Recently, methods like Zero-1-2-3 have focused on single-view based 3D reconstruction and have achieved remarkable success. However, their predictions for unseen areas heavily rely on the inductive bias of large-scale pretrained diffusion models. Although subsequent work, such as DreamComposer, attempts to make predictions more controllable by incorporating additional views, the results remain unrealistic due to feature entanglement in the vanilla latent space, including factors such as lighting, material, and structure. To address these issues, we introduce the Visual Isotropy 3D Reconstruction Model (VI3DRM), a diffusion-based sparse views 3D reconstruction model that operates within an ID consistent and perspective-disentangled 3D latent space. By facilitating the disentanglement of semantic information, color, material properties and lighting, VI3DRM is capable of generating highly realistic images that are indistinguishable from real photographs. By leveraging both real and synthesized images, our approach enables the accurate construction of pointmaps, ultimately producing finely textured meshes or point clouds. On the NVS task, tested on the GSO dataset, VI3DRM significantly outperforms state-of-the-art method DreamComposer, achieving a PSNR of 38.61, an SSIM of 0.929, and an LPIPS of 0.027. Code will be made available upon publication.