Affect-Conditioned Image Generation

TL;DR

This paper presents a method for generating images conditioned on desired affect, using a neural network to estimate affect content from text and images, enabling more nuanced and emotionally expressive image generation.

Contribution

The work introduces a novel affect conditioning approach for text-to-image models, combining affect estimation with generative control to improve creative and expressive outputs.

Findings

Affect conditioning effectively modifies generated images' emotional tone.

The affect estimation model accurately predicts affect content from semantic embeddings.

The method enhances control over generative models for creative applications.

Abstract

In creativity support and computational co-creativity contexts, the task of discovering appropriate prompts for use with text-to-image generative models remains difficult. In many cases the creator wishes to evoke a certain impression with the image, but the task of conferring that succinctly in a text prompt poses a challenge: affective language is nuanced, complex, and model-specific. In this work we introduce a method for generating images conditioned on desired affect, quantified using a psychometrically validated three-component approach, that can be combined with conditioning on text descriptions. We first train a neural network for estimating the affect content of text and images from semantic embeddings, and then demonstrate how this can be used to exert control over a variety of generative models. We show examples of how affect modifies the outputs, provide quantitative and qualitative analysis of its capabilities, and discuss possible extensions and use cases.