GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution

TL;DR

This paper introduces GDPO, a novel reinforcement learning approach for one-step generative image super-resolution, utilizing a noise-aware diffusion model and group preference optimization to improve diversity and detail preservation.

Contribution

The paper proposes GDPO, integrating RL with a noise-aware diffusion model and a group preference strategy for one-step image super-resolution, addressing stochasticity and local detail challenges.

Findings

GDPO enhances super-resolution quality and diversity.

The attribute-aware reward improves local detail preservation.

GDPO outperforms existing methods in experiments.

Abstract

Recently, reinforcement learning (RL) has been employed for improving generative image super-resolution (ISR) performance. However, the current efforts are focused on multi-step generative ISR, while one-step generative ISR remains underexplored due to its limited stochasticity. In addition, RL methods such as Direct Preference Optimization (DPO) require the generation of positive and negative sample pairs offline, leading to a limited number of samples, while Group Relative Policy Optimization (GRPO) only calculates the likelihood of the entire image, ignoring local details that are crucial for ISR. In this paper, we propose Group Direct Preference Optimization (GDPO), a novel approach to integrate RL into one-step generative ISR model training. First, we introduce a noise-aware one-step diffusion model that can generate diverse ISR outputs. To prevent performance degradation caused by noise injection, we introduce an unequal-timestep strategy to decouple the timestep of noise addition from that of diffusion. We then present the GDPO strategy, which integrates the principle of GRPO into DPO, to calculate the group-relative advantage of each online generated sample for model optimization. Meanwhile, an attribute-aware reward function is designed to dynamically evaluate the score of each sample based on its statistics of smooth and texture areas. Experiments demonstrate the effectiveness of GDPO in enhancing the performance of one-step generative ISR models. Code: https://github.com/Joyies/GDPO.