Prompt Evolution for Generative AI: A Classifier-Guided Approach

TL;DR

This paper introduces a novel classifier-guided prompt evolution method that uses evolutionary algorithms to generate diverse outputs aligned with user preferences in generative AI.

Contribution

It presents a multi-objective evolutionary approach leveraging classifiers and pre-trained generative models to improve output relevance and diversity.

Findings

Produces Pareto-optimized images more aligned with user preferences.

Leverages stochastic generative models for implicit mutation operations.

Enhances output diversity and relevance without retraining the generative model.

Abstract

Synthesis of digital artifacts conditioned on user prompts has become an important paradigm facilitating an explosion of use cases with generative AI. However, such models often fail to connect the generated outputs and desired target concepts/preferences implied by the prompts. Current research addressing this limitation has largely focused on enhancing the prompts before output generation or improving the model's performance up front. In contrast, this paper conceptualizes prompt evolution, imparting evolutionary selection pressure and variation during the generative process to produce multiple outputs that satisfy the target concepts/preferences better. We propose a multi-objective instantiation of this broader idea that uses a multi-label image classifier-guided approach. The predicted labels from the classifiers serve as multiple objectives to optimize, with the aim of producing diversified images that meet user preferences. A novelty of our evolutionary algorithm is that the pre-trained generative model gives us implicit mutation operations, leveraging the model's stochastic generative capability to automate the creation of Pareto-optimized images more faithful to user preferences.