Advancing high-fidelity 3D and Texture Generation with 2.5D latents

TL;DR

This paper introduces a unified framework for joint 3D geometry and texture generation using 2.5D latents, leveraging pre-trained 2D models and a novel refiner-decoder to produce high-quality, coherent 3D objects from text and images.

Contribution

The paper presents a new 2.5D representation and a unified generation framework that improves coherence and quality in 3D geometry and texture synthesis.

Findings

Outperforms existing methods in geometry-conditioned texture generation

Generates high-fidelity 3D objects with coherent structure and color

Efficiently transforms 2.5D representations into detailed 3D models

Abstract

Despite the availability of large-scale 3D datasets and advancements in 3D generative models, the complexity and uneven quality of 3D geometry and texture data continue to hinder the performance of 3D generation techniques. In most existing approaches, 3D geometry and texture are generated in separate stages using different models and non-unified representations, frequently leading to unsatisfactory coherence between geometry and texture. To address these challenges, we propose a novel framework for joint generation of 3D geometry and texture. Specifically, we focus in generate a versatile 2.5D representations that can be seamlessly transformed between 2D and 3D. Our approach begins by integrating multiview RGB, normal, and coordinate images into a unified representation, termed as 2.5D latents. Next, we adapt pre-trained 2D foundation models for high-fidelity 2.5D generation, utilizing both text and image conditions. Finally, we introduce a lightweight 2.5D-to-3D refiner-decoder framework that efficiently generates detailed 3D representations from 2.5D images. Extensive experiments demonstrate that our model not only excels in generating high-quality 3D objects with coherent structure and color from text and image inputs but also significantly outperforms existing methods in geometry-conditioned texture generation.