Training-Free Constrained Generation With Stable Diffusion Models

TL;DR

This paper introduces a novel method to incorporate strict physical and domain-specific constraints into stable diffusion models, enhancing their applicability in scientific and engineering tasks involving complex, constrained data generation.

Contribution

The paper presents a new integration of stable diffusion models with constrained optimization, enabling strict adherence to physical and functional constraints in generated outputs.

Findings

Successful material design with precise morphometric properties

Inverse design of materials with specific stress-strain responses

Generation of content under copyright constraints

Abstract

Stable diffusion models represent the state-of-the-art in data synthesis across diverse domains and hold transformative potential for applications in science and engineering, e.g., by facilitating the discovery of novel solutions and simulating systems that are computationally intractable to model explicitly. While there is increasing effort to incorporate physics-based constraints into generative models, existing techniques are either limited in their applicability to latent diffusion frameworks or lack the capability to strictly enforce domain-specific constraints. To address this limitation this paper proposes a novel integration of stable diffusion models with constrained optimization frameworks, enabling the generation of outputs satisfying stringent physical and functional requirements. The effectiveness of this approach is demonstrated through material design experiments requiring adherence to precise morphometric properties, challenging inverse design tasks involving the generation of materials inducing specific stress-strain responses, and copyright-constrained content generation tasks. All code has been released at https://github.com/RAISELab-atUVA/Constrained-Stable-Diffusion.