Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models

TL;DR

This paper investigates the tradeoff between reward optimization and diversity in text-to-image diffusion models, introducing a novel regularization method called Annealed Importance Guidance (AIG) that balances these aspects effectively.

Contribution

The paper proves reward hacking is unavoidable in T2I models, analyzes existing regularization techniques, and proposes AIG as a new inference-time method to optimize reward-diversity tradeoffs.

Findings

AIG retains diversity while optimizing for reward.

AIG achieves Pareto-Optimal reward-diversity balance.

User study confirms AIG improves image diversity and quality.

Abstract

Text-to-image (T2I) diffusion models have become prominent tools for generating high-fidelity images from text prompts. However, when trained on unfiltered internet data, these models can produce unsafe, incorrect, or stylistically undesirable images that are not aligned with human preferences. To address this, recent approaches have incorporated human preference datasets to fine-tune T2I models or to optimize reward functions that capture these preferences. Although effective, these methods are vulnerable to reward hacking, where the model overfits to the reward function, leading to a loss of diversity in the generated images. In this paper, we prove the inevitability of reward hacking and study natural regularization techniques like KL divergence and LoRA scaling, and their limitations for diffusion models. We also introduce Annealed Importance Guidance (AIG), an inference-time regularization inspired by Annealed Importance Sampling, which retains the diversity of the base model while achieving Pareto-Optimal reward-diversity tradeoffs. Our experiments demonstrate the benefits of AIG for Stable Diffusion models, striking the optimal balance between reward optimization and image diversity. Furthermore, a user study confirms that AIG improves diversity and quality of generated images across different model architectures and reward functions.