Fake it till You Make it: Reward Modeling as Discriminative Prediction

TL;DR

This paper introduces GAN-RM, a novel reward modeling framework that uses adversarial discrimination between target and generated samples, reducing reliance on manual annotations and engineering, and improving reinforcement learning for visual generative models.

Contribution

GAN-RM eliminates manual preference annotation and explicit quality dimension engineering by training a reward model through adversarial discrimination with minimal target data.

Findings

Effective in test-time sample filtering for quality selection

Improves post-training reinforcement learning methods like SFT and DPO

Requires only a few hundred target samples for training

Abstract

An effective reward model plays a pivotal role in reinforcement learning for post-training enhancement of visual generative models. However, current approaches of reward modeling suffer from implementation complexity due to their reliance on extensive human-annotated preference data or meticulously engineered quality dimensions that are often incomplete and engineering-intensive. Inspired by adversarial training in generative adversarial networks (GANs), this paper proposes GAN-RM, an efficient reward modeling framework that eliminates manual preference annotation and explicit quality dimension engineering. Our method trains the reward model through discrimination between a small set of representative, unpaired target samples(denoted as Preference Proxy Data) and model-generated ordinary outputs, requiring only a few hundred target samples. Comprehensive experiments demonstrate our GAN-RM's effectiveness across multiple key applications including test-time scaling implemented as Best-of-N sample filtering, post-training approaches like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Code and data will be released at https://github.com/Visualignment/GAN-RM.