Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models
Yaoteng Tan, Zikui Cai, M. Salman Asif

TL;DR
This paper introduces a modular, inference-time safety steering method for text-to-image models that uses foundation models' semantic feedback to enhance safety without degrading image quality.
Contribution
It presents a training-free, energy-based safety control framework that leverages frozen foundation models for scalable, robust safety in generative image models.
Findings
Achieves state-of-the-art robustness against NSFW benchmarks.
Supports multi-target safety steering effectively.
Maintains high image quality on safe prompts.
Abstract
Controlling the behavior of text-to-image generative models is critical for safe and practical deployment. Existing safety approaches typically rely on model fine-tuning or curated datasets, which can degrade generation quality or limit scalability. We propose an inference-time steering framework that leverages gradient feedback from frozen pretrained foundation models to guide the generation process without modifying the underlying generator. Our key observation is that vision-language foundation models encode rich semantic representations that can be repurposed as off-the-shelf supervisory signals during generation. By injecting such feedback through clean latent estimates at each sampling step, our method formulates safety steering as an energy-based sampling problem. This design enables modular, training-free safety control that is compatible with both diffusion and flow-matching…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
