Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models

TL;DR

This paper introduces a modular, inference-time safety steering method for text-to-image models that uses foundation models' semantic feedback to enhance safety without degrading image quality.

Contribution

It presents a training-free, energy-based safety control framework that leverages frozen foundation models for scalable, robust safety in generative image models.

Findings

Achieves state-of-the-art robustness against NSFW benchmarks.

Supports multi-target safety steering effectively.

Maintains high image quality on safe prompts.

Abstract

Controlling the behavior of text-to-image generative models is critical for safe and practical deployment. Existing safety approaches typically rely on model fine-tuning or curated datasets, which can degrade generation quality or limit scalability. We propose an inference-time steering framework that leverages gradient feedback from frozen pretrained foundation models to guide the generation process without modifying the underlying generator. Our key observation is that vision-language foundation models encode rich semantic representations that can be repurposed as off-the-shelf supervisory signals during generation. By injecting such feedback through clean latent estimates at each sampling step, our method formulates safety steering as an energy-based sampling problem. This design enables modular, training-free safety control that is compatible with both diffusion and flow-matching models and can generalize across diverse visual concepts. Experiments demonstrate state-of-the-art robustness against NSFW red-teaming benchmarks and effective multi-target steering, while preserving high generation quality on benign non-targeted prompts. Our framework provides a principled approach for utilizing foundation models as semantic energy estimators, enabling reliable and scalable safety control for text-to-image generation.