LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation

TL;DR

LVRPO introduces a reinforcement learning framework that explicitly aligns language and visual representations, improving multimodal understanding and generation without auxiliary encoders.

Contribution

It proposes a preference-driven reinforcement approach using GRPO for explicit language-visual alignment in multimodal models.

Findings

Outperforms baseline models on multiple multimodal benchmarks.

Enhances both understanding and generation capabilities.

Does not require additional cross-modal loss functions.

Abstract

Unified multimodal pretraining has emerged as a promising paradigm for jointly modeling language and vision within a single foundation model. However, existing approaches largely rely on implicit or indirect alignment signals and remain suboptimal for simultaneously supporting multimodal understanding and generation, particularly in settings that require fine-grained language-visual reasoning and controllable generation. In this work, we propose LVRPO, a language-visual reinforcement-based preference optimization framework that explicitly aligns language and visual representations using Group Relative Policy Optimization (GRPO). Instead of introducing additional alignment losses at the representation level, LVRPO directly optimizes multimodal model behaviors through preference-driven reinforcement signals, encouraging consistent and semantically grounded interactions between language and vision across both understanding and generation tasks. This formulation enables effective alignment without requiring auxiliary encoders or handcrafted cross-modal objectives, and naturally extends to diverse multimodal capabilities. Empirically, LVRPO consistently outperforms strong unified-pretraining baselines on a broad suite of benchmarks spanning multimodal understanding, generation, and reasoning.