Threshold-Guided Optimization for Visual Generative Models

TL;DR

This paper introduces a threshold-guided optimization method for aligning visual generative models with scalar feedback, replacing pairwise comparisons with a data-driven threshold approach for improved scalability and efficiency.

Contribution

It proposes a novel threshold-guided alignment framework that enables direct optimization from scalar feedback, improving preference alignment without relying on paired comparisons.

Findings

Consistently improves preference alignment across multiple test sets and reward models.

Enables effective optimization directly from scalar feedback, reducing reliance on annotated pairs.

Incorporates confidence weighting to enhance sample efficiency.

Abstract

Aligning large visual generative models with human feedback is often performed through pairwise preference optimization. While such approaches are conceptually simple, they fundamentally rely on annotated pairs, limiting scalability in settings where feedback is collected as independent scalar ratings. In this work, we revisit the KL-regularized alignment objective and show that the optimal policy implicitly compares each sample's reward to an instance-specific baseline that is generally intractable. We propose a threshold-guided alignment framework that replaces this oracle baseline with a data-driven global threshold estimated from empirical score statistics. This formulation turns alignment into a binary decision task on unpaired data, enabling effective optimization directly from scalar feedback. We also incorporate a confidence weighting term to emphasize samples whose scores deviate strongly from the threshold, improving sample efficiency. Experiments across both diffusion and masked generative paradigms, spanning three test sets and five reward models, show that our method consistently improves preference alignment over previous methods. These results position our threshold-guided framework as a simple yet principled alternative for aligning visual generative models without paired comparisons.