WorldMesh: Generating Navigable Multi-Room 3D Scenes via Mesh-Conditioned Image Diffusion

TL;DR

This paper introduces WorldMesh, a geometry-first method for generating large-scale, immersive 3D scenes by combining mesh-based structural scaffolds with conditioned image synthesis for realistic appearance.

Contribution

It proposes a novel mesh-conditioned image diffusion approach that decouples scene structure from appearance, enabling scalable and consistent 3D scene generation.

Findings

Enables generation of large, diverse 3D scenes with high object richness.

Maintains scene and object consistency at large scales.

Produces photorealistic, environment-scale 3D worlds.

Abstract

Recent progress in image and video synthesis has inspired their use in advancing 3D scene generation. However, we observe that text-to-image and -video approaches struggle to maintain scene- and object-level consistency beyond a limited environment scale due to the absence of explicit geometry. We thus present a geometry-first approach that decouples this complex problem of large-scale 3D scene synthesis into its structural composition, represented as a mesh scaffold, and realistic appearance synthesis, which leverages powerful image synthesis models conditioned on the mesh scaffold. From an input text description, we first construct a mesh capturing the environment's geometry (walls, floors, etc.), and then use image synthesis, segmentation and object reconstruction to populate the mesh structure with objects in realistic layouts. This mesh scaffold is then rendered to condition image synthesis, providing a structural backbone for consistent appearance generation. This enables scalable, arbitrarily-sized 3D scenes of high object richness and diversity, combining robust 3D consistency with photorealistic detail. We believe this marks a significant step toward generating truly environment-scale, immersive 3D worlds.