Evolutionary Optimization Trumps Adam Optimization on Embedding Space Exploration

TL;DR

This paper demonstrates that a gradient-free evolutionary optimizer, sep-CMA-ES, outperforms Adam in prompt-embedding search for diffusion models, enhancing image quality and alignment without model fine-tuning.

Contribution

It introduces the use of sep-CMA-ES for prompt-embedding optimization in diffusion models, showing superior performance over Adam in various trade-offs.

Findings

sep-CMA-ES achieves higher objective scores than Adam across prompts.

Evolutionary optimization improves aesthetics-alignment trade-offs.

Resource usage for sep-CMA-ES is comparable or better than Adam.

Abstract

Deep diffusion models have revolutionized image generation by producing high-quality outputs. However, achieving specific objectives with these models often requires costly adaptations such as fine-tuning, which can be resource-intensive and time-consuming. An alternative approach is inference-time control, which involves optimizing the prompt embeddings to guide the generation process without altering the model weights. We explore prompt-embedding search optimization for the Stable Diffusion XL Turbo model, comparing a gradient-free evolutionary approach, the Separable Covariance Matrix Adaptation Evolution Strategy (sep-CMA-ES), against the widely used gradient-based optimizer Adaptive Moment Estimation (Adam). Candidate images are evaluated by a weighted objective that combines LAION Aesthetic Predictor V2 and CLIPScore, enabling explicit trade-offs between aesthetic quality and prompt-image alignment. On 36 prompts sampled from Parti Prompts (P2) under three weight settings (aesthetics-only, balanced, alignment-only), sep-CMA-ES consistently achieves higher objective values than Adam. We additionally analyze divergence from the unoptimized baseline using cosine similarity and SSIM and report the compute and memory footprints. These results suggest that sep-CMA-ES is an effective inference-time optimizer for prompt-embedding search, improving aesthetics-alignment trade-offs and resource usage without model fine-tuning.