ITI-GEN: Inclusive Text-to-Image Generation

TL;DR

ITI-GEN introduces a novel method for inclusive text-to-image generation that uses reference images and learned prompt embeddings to better represent diverse attributes without fine-tuning, improving inclusivity in generated images.

Contribution

The paper proposes ITI-GEN, a reference-image-based approach that enhances inclusivity in text-to-image models without requiring model fine-tuning.

Findings

Significantly improves diversity of generated images across attributes.

Effectively uses reference images to represent hard-to-describe categories.

Achieves state-of-the-art performance in inclusive image generation.

Abstract

Text-to-image generative models often reflect the biases of the training data, leading to unequal representations of underrepresented groups. This study investigates inclusive text-to-image generative models that generate images based on human-written prompts and ensure the resulting images are uniformly distributed across attributes of interest. Unfortunately, directly expressing the desired attributes in the prompt often leads to sub-optimal results due to linguistic ambiguity or model misrepresentation. Hence, this paper proposes a drastically different approach that adheres to the maxim that "a picture is worth a thousand words". We show that, for some attributes, images can represent concepts more expressively than text. For instance, categories of skin tones are typically hard to specify by text but can be easily represented by example images. Building upon these insights, we propose a novel approach, ITI-GEN, that leverages readily available reference images for Inclusive Text-to-Image GENeration. The key idea is learning a set of prompt embeddings to generate images that can effectively represent all desired attribute categories. More importantly, ITI-GEN requires no model fine-tuning, making it computationally efficient to augment existing text-to-image models. Extensive experiments demonstrate that ITI-GEN largely improves over state-of-the-art models to generate inclusive images from a prompt. Project page: https://czhang0528.github.io/iti-gen.