MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation

TL;DR

This paper introduces MAR-GRPO, a stabilized reinforcement learning framework for hybrid autoregressive-diffusion image generation, reducing gradient noise and improving stability and quality.

Contribution

It proposes multi-trajectory expectation and token uncertainty estimation to stabilize training and enhance image generation quality in MAR models.

Findings

Improved visual quality over baseline models

Enhanced training stability and convergence

Better spatial structure understanding in generated images

Abstract

Reinforcement learning (RL) has been successfully applied to autoregressive (AR) and diffusion models. However, extending RL to hybrid AR-diffusion frameworks remains challenging due to interleaved inference and noisy log-probability estimation. In this work, we study masked autoregressive models (MAR) and show that the diffusion head plays a critical role in training dynamics, often introducing noisy gradients that lead to instability and early performance saturation. To address this issue, we propose a stabilized RL framework for MAR. We introduce multi-trajectory expectation (MTE), which estimates the optimization direction by averaging over multiple diffusion trajectories, thereby reducing diffusion-induced gradient noise. To avoid over-smoothing, we further estimate token-wise uncertainty from multiple trajectories and apply multi-trajectory optimization only to the top-k% uncertain tokens. In addition, we introduce a consistency-aware token selection strategy that filters out AR tokens that are less aligned with the final generated content. Extensive experiments across multiple benchmarks demonstrate that our method consistently improves visual quality, training stability, and spatial structure understanding over baseline GRPO and pre-RL models. Code is available at: https://github.com/AMAP-ML/mar-grpo.