LDM: Large Tensorial SDF Model for Textured Mesh Generation

TL;DR

LDM introduces a novel framework that generates high-fidelity textured 3D meshes from single images or text prompts, using multi-view diffusion, tensorial SDF prediction, and mesh refinement for rapid, high-quality asset creation.

Contribution

The paper presents a new feed-forward approach combining diffusion models and tensorial SDF prediction for textured mesh generation from minimal input.

Findings

Generates diverse, high-quality textured meshes within seconds.

Outperforms existing methods in mesh quality and rendering suitability.

Effectively decouples illumination in textured mesh generation.

Abstract

Previous efforts have managed to generate production-ready 3D assets from text or images. However, these methods primarily employ NeRF or 3D Gaussian representations, which are not adept at producing smooth, high-quality geometries required by modern rendering pipelines. In this paper, we propose LDM, a novel feed-forward framework capable of generating high-fidelity, illumination-decoupled textured mesh from a single image or text prompts. We firstly utilize a multi-view diffusion model to generate sparse multi-view inputs from single images or text prompts, and then a transformer-based model is trained to predict a tensorial SDF field from these sparse multi-view image inputs. Finally, we employ a gradient-based mesh optimization layer to refine this model, enabling it to produce an SDF field from which high-quality textured meshes can be extracted. Extensive experiments demonstrate that our method can generate diverse, high-quality 3D mesh assets with corresponding decomposed RGB textures within seconds.