Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

TL;DR

This paper presents LLaVA-Reward, a novel reward model utilizing pretrained multimodal large language models to evaluate text-to-image generation quality efficiently across multiple perspectives, improving over existing methods.

Contribution

Introduction of LLaVA-Reward, a reward model that directly uses MLLM hidden states and a SkipCA module for better text-image correlation, supporting various preference data types for improved evaluation.

Findings

LLaVA-Reward outperforms existing methods in automatic evaluation accuracy.

It effectively scales inference-time evaluation for text-to-image generation.

The model enhances text-image correlation reasoning with the SkipCA module.

Abstract

We introduce LLaVA-Reward, an efficient reward model designed to automatically evaluate text-to-image (T2I) generations across multiple perspectives, leveraging pretrained multimodal large language models (MLLMs). Existing MLLM-based approaches require instruction-following data for supervised fine-tuning and evaluate generation quality on analyzing text response, which is time-consuming and difficult to train. To address this problem, we propose LLaVA-Reward, which directly utilizes the hidden states of MLLMs given text-image pairs. To enhance the bidirectional interaction between visual and textual representations in decoder-only MLLMs, we further propose adding a Skip-connection Cross Attention (SkipCA) module. This design enhances text-image correlation reasoning by connecting early-layer visual features with later-layer hidden representations. In addition, LLaVA-Reward supports different types of preference data for efficient fine-tuning, including paired preference data and unpaired data. We train LLaVA-Reward on four evaluation perspectives: text-image alignment, fidelity/artifact, safety, and overall ranking. Empirical results demonstrate that LLaVA-Reward outperforms conventional and MLLM-based methods in generating human-aligned scores for automatic evaluations and inference-time scaling in text-to-image generations.