Latent Guard: a Safety Framework for Text-to-image Generation

TL;DR

Latent Guard introduces a novel safety framework for text-to-image models that detects harmful concepts in input text by analyzing the latent space of the text encoder, enhancing safety without extensive dataset training.

Contribution

It proposes a new latent space-based safety mechanism for T2I models, utilizing contrastive learning and data generation to improve harmful content detection.

Findings

Effective in identifying harmful concepts across three datasets

Outperforms four baseline safety methods

Provides a flexible, scalable safety solution for T2I models

Abstract

With the ability to generate high-quality images, text-to-image (T2I) models can be exploited for creating inappropriate content. To prevent misuse, existing safety measures are either based on text blacklists, which can be easily circumvented, or harmful content classification, requiring large datasets for training and offering low flexibility. Hence, we propose Latent Guard, a framework designed to improve safety measures in text-to-image generation. Inspired by blacklist-based approaches, Latent Guard learns a latent space on top of the T2I model's text encoder, where it is possible to check the presence of harmful concepts in the input text embeddings. Our proposed framework is composed of a data generation pipeline specific to the task using large language models, ad-hoc architectural components, and a contrastive learning strategy to benefit from the generated data. The effectiveness of our method is verified on three datasets and against four baselines. Code and data will be shared at https://latentguard.github.io/.