Extend3D: Town-Scale 3D Generation

TL;DR

Extend3D introduces a novel, training-free pipeline for generating large-scale 3D scenes from a single image by extending and dividing the latent space, refining patches, and optimizing for better structure and texture fidelity.

Contribution

The paper presents a new method that extends object-centric 3D generative models to scene-scale generation without training, using patch-wise generation and 3D-aware optimization.

Findings

Outperforms prior methods in human preference tests.

Achieves higher quantitative scores in 3D scene generation.

Effectively completes 3D structures via under-noising technique.

Abstract

In this paper, we propose Extend3D, a training-free pipeline for 3D scene generation from a single image, built upon an object-centric 3D generative model. To overcome the limitations of fixed-size latent spaces in object-centric models for representing wide scenes, we extend the latent space in the $x$ and $y$ directions. Then, by dividing the extended latent space into overlapping patches, we apply the object-centric 3D generative model to each patch and couple them at each time step. Since patch-wise 3D generation with image conditioning requires strict spatial alignment between image and latent patches, we initialize the scene using a point cloud prior from a monocular depth estimator and iteratively refine occluded regions through SDEdit. We discovered that treating the incompleteness of 3D structure as noise during 3D refinement enables 3D completion via a concept, which we term under-noising. Furthermore, to address the sub-optimality of object-centric models for sub-scene generation, we optimize the extended latent during denoising, ensuring that the denoising trajectories remain consistent with the sub-scene dynamics. To this end, we introduce 3D-aware optimization objectives for improved geometric structure and texture fidelity. We demonstrate that our method yields better results than prior methods, as evidenced by human preference and quantitative experiments.