GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusion

TL;DR

GTA introduces a two-stage geometry-then-appearance framework for improved 3D scene generation from a single image, enhancing structural fidelity and cross-view consistency using dedicated diffusion models.

Contribution

The paper proposes a novel Geometry-Then-Appearance paradigm with a two-stage diffusion approach and training strategies that significantly improve 3D scene fidelity and versatility.

Findings

Outperforms existing methods in fidelity, visual quality, and geometric accuracy.

Enhances existing image-to-3D pipelines as a general improvement module.

Demonstrates broad applicability and data efficiency in downstream tasks.

Abstract

Recent developments in generative models and large-scale datasets have substantially advanced 3D world generation, facilitating a broad range of domains including spatial intelligence, embodied intelligence, and autonomous driving. While achieving remarkable progress, existing approaches to 3D world generation typically prioritize appearance prediction with limited modeling of the underlying geometry, leading to issues such as unreliable scene structure estimation and degraded cross-view consistency. To address these limitations, motivated by the coarse-to-fine nature of human visual perception, we propose GTA, a novel image-to-3D world generation method following a Geometry-Then-Appearance paradigm. Specifically, given a single input image, to improve the structural fidelity of synthesized 3D scenes, GTA adopts a two-stage framework with two dedicated video diffusion models, which first generate coarse geometric structure from novel viewpoints and then synthesize fine-grained appearance conditioned on the predicted geometry. To further enhance cross-view appearance consistency, we introduce a random latent shuffle strategy during the training process, along with a test-time scaling scheme that improves perceptual quality without compromising quantitative performance. Extensive experiments have demonstrated that our proposed method consistently outperforms existing approaches in terms of fidelity, visual quality, and geometric accuracy. Moreover, GTA is shown to be effective as a general enhancement module that further improves the generation quality of existing image-to-3D world pipelines, as well as supporting multiple downstream applications and exhibiting favorable data efficiency during model training, highlighting its versatility and broad applicability. Project page: https://hanxinzhu-lab.github.io/GTA/.