GraphicsDreamer: Image to 3D Generation with Physical Consistency

TL;DR

GraphicsDreamer is a novel method that generates high-quality, physically consistent 3D meshes from single images, integrating PBR lighting and topology optimization to meet industrial standards.

Contribution

It introduces a cross-domain diffusion model with PBR integration for detailed 3D asset creation from single images, including topology optimization and UV unwrapping.

Findings

Produces high-quality 3D assets with realistic textures and relighting.

Ensures physical consistency through PBR constraints.

Operates efficiently compared to previous methods.

Abstract

Recently, the surge of efficient and automated 3D AI-generated content (AIGC) methods has increasingly illuminated the path of transforming human imagination into complex 3D structures. However, the automated generation of 3D content is still significantly lags in industrial application. This gap exists because 3D modeling demands high-quality assets with sharp geometry, exquisite topology, and physically based rendering (PBR), among other criteria. To narrow the disparity between generated results and artists' expectations, we introduce GraphicsDreamer, a method for creating highly usable 3D meshes from single images. To better capture the geometry and material details, we integrate the PBR lighting equation into our cross-domain diffusion model, concurrently predicting multi-view color, normal, depth images, and PBR materials. In the geometry fusion stage, we continue to enforce the PBR constraints, ensuring that the generated 3D objects possess reliable texture details, supporting realistic relighting. Furthermore, our method incorporates topology optimization and fast UV unwrapping capabilities, allowing the 3D products to be seamlessly imported into graphics engines. Extensive experiments demonstrate that our model can produce high quality 3D assets in a reasonable time cost compared to previous methods.