Bias at the End of the Score

TL;DR

This paper audits reward models used in text-to-image systems, revealing they encode demographic biases that affect model outputs and challenge their reliability as quality metrics.

Contribution

It provides the first large-scale analysis of demographic biases in reward models, demonstrating their impact on model behavior and diversity.

Findings

Reward models encode demographic biases.

Biases lead to sexualization and stereotype reinforcement.

Reward-guided optimization collapses demographic diversity.

Abstract

Reward models (RMs) are inherently non-neutral value functions designed and trained to encode specific objectives, such as human preferences or text-image alignment. RMs have become crucial components of text-to-image (T2I) generation systems where they are used at various stages for dataset filtering, as evaluation metrics, as a supervisory signal during optimization of parameters, and for post-generation safety and quality filtering of T2I outputs. While specific problems with the integration of RMs into the T2I pipeline have been studied (e.g. reward hacking or mode collapse), their robustness and fairness as scoring functions remains largely unknown. We conduct a large scale audit of RM robustness with respect to demographic biases during T2I model training and generation. We provide quantitative and qualitative evidence that while originally developed as quality measures, RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects reinforce gender/racial stereotypes, and collapse demographic diversity. These findings highlight shortcomings in current reward models, challenge their reliability as quality metrics, and underscore the need for improved data collection and training procedures to enable more robust scoring.