AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning

TL;DR

This paper introduces AR-GRPO, a novel reinforcement learning framework that enhances autoregressive image generation models by optimizing for multiple quality metrics, leading to higher quality and more preferred images.

Contribution

AR-GRPO adapts the Group Relative Policy Optimization algorithm for autoregressive image models, integrating RL to improve image quality and fidelity beyond standard methods.

Findings

Significant improvements in image quality metrics.

Enhanced human preference for generated images.

Effective RL-based optimization for AR image models.

Abstract

Inspired by the success of reinforcement learning (RL) in refining large language models (LLMs), we propose AR-GRPO, an approach to integrate online RL training into autoregressive (AR) image generation models. We adapt the Group Relative Policy Optimization (GRPO) algorithm to refine the vanilla autoregressive models' outputs by carefully designed reward functions that evaluate generated images across multiple quality dimensions, including perceptual quality, realism, and semantic fidelity. We conduct comprehensive experiments on both class-conditional (i.e., class-to-image) and text-conditional (i.e., text-to-image) image generation tasks, demonstrating that our RL-enhanced framework significantly improves both the image quality and human preference of generated images compared to the standard AR baselines. Our results show consistent improvements across various evaluation metrics, establishing the viability of RL-based optimization for AR image generation and opening new avenues for controllable and high-quality image synthesis. The source codes and models are available at: https://github.com/Kwai-Klear/AR-GRPO.