Sketch-Based Facade Renovation With Generative AI: A Streamlined Framework for Bypassing As-Built Modelling in Industrial Adaptive Reuse

TL;DR

This paper introduces a novel AI-driven framework that streamlines facade renovation design by directly transforming rough sketches and textual descriptions into detailed, photorealistic proposals, bypassing traditional detailed as-built modelling.

Contribution

The paper presents a three-stage AI framework combining vision-language models, diffusion models, and inpainting to generate facade renovation proposals from minimal input, reducing time and effort.

Findings

Framework effectively preserves original structure while enhancing detail quality.

Enables rapid exploration and iteration of design alternatives.

Demonstrates success on datasets and real industrial buildings.

Abstract

Facade renovation offers a more sustainable alternative to full demolition, yet producing design proposals that preserve existing structures while expressing new intent remains challenging. Current workflows typically require detailed as-built modelling before design, which is time-consuming, labour-intensive, and often involves repeated revisions. To solve this issue, we propose a three-stage framework combining generative artificial intelligence (AI) and vision-language models (VLM) that directly processes rough structural sketch and textual descriptions to produce consistent renovation proposals. First, the input sketch is used by a fine-tuned VLM model to predict bounding boxes specifying where modifications are needed and which components should be added. Next, a stable diffusion model generates detailed sketches of new elements, which are merged with the original outline through a generative inpainting pipeline. Finally, ControlNet is employed to refine the result into a photorealistic image. Experiments on datasets and real industrial buildings indicate that the proposed framework can generate renovation proposals that preserve the original structure while improving facade detail quality. This approach effectively bypasses the need for detailed as-built modelling, enabling architects to rapidly explore design alternatives, iterate on early-stage concepts, and communicate renovation intentions with greater clarity.