MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion

TL;DR

MeSS introduces a novel pipeline that combines enhanced image diffusion models with geometric priors and control mechanisms to generate high-quality, cross-view consistent outdoor city scenes from mesh models, improving realism and style diversity.

Contribution

The paper presents a new method that improves cross-view consistency in outdoor scene generation by integrating control-based diffusion models with geometric priors and scene reconstruction.

Findings

Outperforms existing methods in geometric alignment.

Produces high-quality, style-consistent outdoor scenes.

Enables diverse scene rendering through relighting and style transfer.

Abstract

Mesh models have become increasingly accessible for numerous cities; however, the lack of realistic textures restricts their application in virtual urban navigation and autonomous driving. To address this, this paper proposes MeSS (Meshbased Scene Synthesis) for generating high-quality, styleconsistent outdoor scenes with city mesh models serving as the geometric prior. While image and video diffusion models can leverage spatial layouts (such as depth maps or HD maps) as control conditions to generate street-level perspective views, they are not directly applicable to 3D scene generation. Video diffusion models excel at synthesizing consistent view sequences that depict scenes but often struggle to adhere to predefined camera paths or align accurately with rendered control videos. In contrast, image diffusion models, though unable to guarantee cross-view visual consistency, can produce more geometry-aligned results when combined with ControlNet. Building on this insight, our approach enhances image diffusion models by improving cross-view consistency. The pipeline comprises three key stages: first, we generate geometrically consistent sparse views using Cascaded Outpainting ControlNets; second, we propagate denser intermediate views via a component dubbed AGInpaint; and third, we globally eliminate visual inconsistencies (e.g., varying exposure) using the GCAlign module. Concurrently with generation, a 3D Gaussian Splatting (3DGS) scene is reconstructed by initializing Gaussian balls on the mesh surface. Our method outperforms existing approaches in both geometric alignment and generation quality. Once synthesized, the scene can be rendered in diverse styles through relighting and style transfer techniques. project page: https://albertchen98.github.io/mess/