SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards

TL;DR

SUDER introduces a self-supervised dual reward framework that leverages the inverse relationship between understanding and generation tasks to improve large multimodal models' alignment and performance without external supervision.

Contribution

The paper proposes a novel dual self-reward mechanism exploiting the inverse nature of understanding and generation tasks to enhance multimodal models.

Findings

Significant improvement in text-to-image generation quality.

Enhanced vision-language alignment without external supervision.

Effective self-improvement of multimodal models through dual rewards.

Abstract

Building upon large language models (LLMs), recent large multimodal models (LMMs) unify cross-model understanding and generation into a single framework. However, LMMs still struggle to achieve accurate vision-language alignment, prone to generating text responses contradicting the visual input or failing to follow the text-to-image prompts. Current solutions require external supervision (e.g., human feedback or reward models) and only address unidirectional tasks-either understanding or generation. In this work, based on the observation that understanding and generation are naturally inverse dual tasks, we propose \textbf{SUDER} (\textbf{S}elf-improving \textbf{U}nified LMMs with \textbf{D}ual s\textbf{E}lf-\textbf{R}ewards), a framework reinforcing the understanding and generation capabilities of LMMs with a self-supervised dual reward mechanism. SUDER leverages the inherent duality between understanding and generation tasks to provide self-supervised optimization signals for each other. Specifically, we sample multiple outputs for a given input in one task domain, then reverse the input-output pairs to compute the dual likelihood within the model as self-rewards for optimization. Extensive experimental results on visual understanding and generation benchmarks demonstrate that our method can effectively enhance the performance of the model without any external supervision, especially achieving remarkable improvements in text-to-image tasks.