Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

TL;DR

This paper introduces a reinforcement learning strategy to enable existing unified vision-language models to generate coherent multimodal interleaved outputs, crucial for tasks like visual storytelling, without requiring large-scale interleaved datasets.

Contribution

It proposes a novel policy optimization framework extending GRPO to multimodal generation, incorporating hybrid rewards and process-level guidance to improve interleaved output quality.

Findings

Significant improvement in interleaved generation quality and coherence.

Effective training with hybrid rewards and process-level guidance.

Validated on MMIE and InterleavedBench datasets.

Abstract

Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.