DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation

TL;DR

DreamMesh introduces a novel 3D generation approach that explicitly models triangle meshes and textures, resulting in high-fidelity, textured 3D models from text prompts, overcoming limitations of implicit NeRF representations.

Contribution

It presents a coarse-to-fine architecture that jointly manipulates meshes and textures, enabling high-quality, textured 3D models without tuning, outperforming existing text-to-3D methods.

Findings

Outperforms state-of-the-art methods in fidelity and detail

Produces textured 3D models with consistent geometry

Uses a tuning-free, multi-view texturing approach

Abstract

Learning radiance fields (NeRF) with powerful 2D diffusion models has garnered popularity for text-to-3D generation. Nevertheless, the implicit 3D representations of NeRF lack explicit modeling of meshes and textures over surfaces, and such surface-undefined way may suffer from the issues, e.g., noisy surfaces with ambiguous texture details or cross-view inconsistency. To alleviate this, we present DreamMesh, a novel text-to-3D architecture that pivots on well-defined surfaces (triangle meshes) to generate high-fidelity explicit 3D model. Technically, DreamMesh capitalizes on a distinctive coarse-to-fine scheme. In the coarse stage, the mesh is first deformed by text-guided Jacobians and then DreamMesh textures the mesh with an interlaced use of 2D diffusion models in a tuning free manner from multiple viewpoints. In the fine stage, DreamMesh jointly manipulates the mesh and refines the texture map, leading to high-quality triangle meshes with high-fidelity textured materials. Extensive experiments demonstrate that DreamMesh significantly outperforms state-of-the-art text-to-3D methods in faithfully generating 3D content with richer textual details and enhanced geometry. Our project page is available at https://dreammesh.github.io.