Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation

TL;DR

This paper examines the potential and limitations of fine-grained human feedback for improving reward models in text-to-image generation, highlighting complexities and conditions under which it outperforms coarse feedback.

Contribution

It provides an empirical analysis of fine-grained versus coarse feedback, revealing challenges and conditions affecting their effectiveness in reward modeling for text-to-image tasks.

Findings

Fine-grained feedback can worsen models with limited budgets in some cases.

In controlled settings, fine-grained rewards outperform coarse feedback.

Model choice and feedback alignment critically influence outcomes.

Abstract

Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback which captures nuanced distinctions in image quality and prompt-alignment, compared to traditional coarse-grained feedback (for example, thumbs up/down or ranking between a set of options). While fine-grained feedback holds promise, particularly for systems catering to diverse societal preferences, we show that demonstrating its superiority to coarse-grained feedback is not automatic. Through experiments on real and synthetic preference data, we surface the complexities of building effective models due to the interplay of model choice, feedback type, and the alignment between human judgment and computational interpretation. We identify key challenges in eliciting and utilizing fine-grained feedback, prompting a reassessment of its assumed benefits and practicality. Our findings -- e.g., that fine-grained feedback can lead to worse models for a fixed budget, in some settings; however, in controlled settings with known attributes, fine grained rewards can indeed be more helpful -- call for careful consideration of feedback attributes and potentially beckon novel modeling approaches to appropriately unlock the potential value of fine-grained feedback in-the-wild.