Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generation

TL;DR

This paper introduces Bayesian Prior-Guided Optimization (BPGO), an extension of GRPO that models reward uncertainty to improve visual generation quality, semantic alignment, and convergence speed.

Contribution

BPGO explicitly models reward uncertainty with a semantic prior, improving over GRPO by better handling ambiguous feedback in visual generative models.

Findings

BPGO outperforms GRPO in semantic alignment and perceptual fidelity.

BPGO achieves faster convergence in image and video generation.

BPGO effectively manages reward ambiguity through Bayesian trust modulation.

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an effective and lightweight framework for post-training visual generative models. However, its performance is fundamentally limited by the ambiguity of textual visual correspondence: a single prompt may validly describe diverse visual outputs, and a single image or video may support multiple equally correct interpretations. This many to many relationship leads reward models to generate uncertain and weakly discriminative signals, causing GRPO to underutilize reliable feedback and overfit noisy ones. We introduce Bayesian Prior-Guided Optimization (BPGO), a novel extension of GRPO that explicitly models reward uncertainty through a semantic prior anchor. BPGO adaptively modulates optimization trust at two levels: inter-group Bayesian trust allocation emphasizes updates from groups consistent with the prior while down-weighting ambiguous ones, and intra-group prior-anchored renormalization sharpens sample distinctions by expanding confident deviations and compressing uncertain scores. Across both image and video generation tasks, BPGO delivers consistently stronger semantic alignment, enhanced perceptual fidelity, and faster convergence than standard GRPO and recent variants.