Parameter-aware high-fidelity microstructure generation using stable diffusion

TL;DR

This paper introduces a novel process-aware microstructure generation method using a fine-tuned Stable Diffusion model with numeric-aware embeddings, enabling controlled, realistic microstructure synthesis conditioned on process parameters.

Contribution

It adapts the SD3.5-Large diffusion model for microstructure generation by incorporating continuous process variables and fine-tuning with LoRA, addressing data scarcity and enabling controlled synthesis.

Findings

Achieved 97.1% accuracy and 85.7% IoU in realism validation.

Two-point correlation and lineal-path errors below 2.1% and 0.6%.

First adaptation of SD3.5-Large for process-aware microstructure generation.

Abstract

Synthesizing realistic microstructure images conditioned on processing parameters is crucial for understanding process-structure relationships in materials design. However, this task remains challenging due to limited training micrographs and the continuous nature of processing variables. To overcome these challenges, we present a novel process-aware generative modeling approach based on Stable Diffusion 3.5 Large (SD3.5-Large), a state-of-the-art text-to-image diffusion model adapted for microstructure generation. Our method introduces numeric-aware embeddings that encode continuous variables (annealing temperature, time, and magnification) directly into the model's conditioning, enabling controlled image generation under specified process conditions and capturing process-driven microstructural variations. To address data scarcity and computational constraints, we fine-tune only a small fraction of the model's weights via DreamBooth and Low-Rank Adaptation (LoRA), efficiently transferring the pre-trained model to the materials domain. We validate realism using a semantic segmentation model based on a fine-tuned U-Net with a VGG16 encoder on 24 labeled micrographs. It achieves 97.1% accuracy and 85.7% mean IoU, outperforming previous methods. Quantitative analyses using physical descriptors and spatial statistics show strong agreement between synthetic and real microstructures. Specifically, two-point correlation and lineal-path errors remain below 2.1% and 0.6%, respectively. Our method represents the first adaptation of SD3.5-Large for process-aware microstructure generation, offering a scalable approach for data-driven materials design.