Text Semantics to Image Generation: A method of building facades design base on Stable Diffusion model

TL;DR

This paper enhances architectural facade image generation by fine-tuning Stable Diffusion with LoRA and applying ControlNet to improve controllability based on textual architectural descriptions.

Contribution

It introduces a combined multi-network approach using LoRA and ControlNet to improve controllability and efficiency in text-to-facade image generation.

Findings

LoRA significantly reduces fine-tuning complexity.

ControlNet increases controllability of generated images.

The method effectively generates architectural facades based on text descriptions.

Abstract

Stable Diffusion model has been extensively employed in the study of archi-tectural image generation, but there is still an opportunity to enhance in terms of the controllability of the generated image content. A multi-network combined text-to-building facade image generating method is proposed in this work. We first fine-tuned the Stable Diffusion model on the CMP Fa-cades dataset using the LoRA (Low-Rank Adaptation) approach, then we ap-ply the ControlNet model to further control the output. Finally, we contrast-ed the facade generating outcomes under various architectural style text con-tents and control strategies. The results demonstrate that the LoRA training approach significantly decreases the possibility of fine-tuning the Stable Dif-fusion large model, and the addition of the ControlNet model increases the controllability of the creation of text to building facade images. This pro-vides a foundation for subsequent studies on the generation of architectural images.