TerraFusion: Joint Generation of Terrain Geometry and Texture Using Latent Diffusion Models

TL;DR

This paper introduces TerraFusion, a novel method that jointly generates correlated terrain heightmaps and textures using latent diffusion models, enabling realistic and controllable 3D terrain creation.

Contribution

It presents a joint generation framework with unsupervised training and supervised user control, addressing the lack of correlation modeling in existing terrain generation methods.

Findings

Allows intuitive terrain generation with user sketches

Preserves correlation between heightmaps and textures

Produces realistic 3D terrain models

Abstract

3D terrain models are essential in fields such as video game development and film production. Since surface color often correlates with terrain geometry, capturing this relationship is crucial to achieving realism. However, most existing methods generate either a heightmap or a texture, without sufficiently accounting for the inherent correlation. In this paper, we propose a method that jointly generates terrain heightmaps and textures using a latent diffusion model. First, we train the model in an unsupervised manner to randomly generate paired heightmaps and textures. Then, we perform supervised learning of an external adapter to enable user control via hand-drawn sketches. Experiments show that our approach allows intuitive terrain generation while preserving the correlation between heightmaps and textures.