Aligning Text-to-Image Models using Human Feedback

TL;DR

This paper introduces a human feedback-based fine-tuning approach for text-to-image models, significantly enhancing their alignment with textual prompts by leveraging human evaluations and reward modeling.

Contribution

It presents a novel three-stage method for aligning text-to-image models using human feedback, including feedback collection, reward function training, and model fine-tuning.

Findings

Improved accuracy in generating specified object attributes

Effective use of human feedback for model alignment

Analysis of design choices affecting alignment-fidelity tradeoffs

Abstract

Deep generative models have shown impressive results in text-to-image synthesis. However, current text-to-image models often generate images that are inadequately aligned with text prompts. We propose a fine-tuning method for aligning such models using human feedback, comprising three stages. First, we collect human feedback assessing model output alignment from a set of diverse text prompts. We then use the human-labeled image-text dataset to train a reward function that predicts human feedback. Lastly, the text-to-image model is fine-tuned by maximizing reward-weighted likelihood to improve image-text alignment. Our method generates objects with specified colors, counts and backgrounds more accurately than the pre-trained model. We also analyze several design choices and find that careful investigations on such design choices are important in balancing the alignment-fidelity tradeoffs. Our results demonstrate the potential for learning from human feedback to significantly improve text-to-image models.