MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models

TL;DR

MMIG-Bench is a comprehensive and explainable benchmark for evaluating multi-modal image generation models, integrating diverse tasks and introducing new metrics to better assess quality, alignment, and human preferences.

Contribution

It introduces MMIG-Bench, a unified multi-modal image generation benchmark with novel evaluation metrics, including an Aspect Matching Score, and provides extensive benchmarking of 17 models.

Findings

The Aspect Matching Score correlates strongly with human judgments.

MMIG-Bench reveals strengths and weaknesses of current models.

Benchmarking results offer insights into architecture and data impacts.

Abstract

Recent multimodal image generators such as GPT-4o, Gemini 2.0 Flash, and Gemini 2.5 Pro excel at following complex instructions, editing images and maintaining concept consistency. However, they are still evaluated by disjoint toolkits: text-to-image (T2I) benchmarks that lacks multi-modal conditioning, and customized image generation benchmarks that overlook compositional semantics and common knowledge. We propose MMIG-Bench, a comprehensive Multi-Modal Image Generation Benchmark that unifies these tasks by pairing 4,850 richly annotated text prompts with 1,750 multi-view reference images across 380 subjects, spanning humans, animals, objects, and artistic styles. MMIG-Bench is equipped with a three-level evaluation framework: (1) low-level metrics for visual artifacts and identity preservation of objects; (2) novel Aspect Matching Score (AMS): a VQA-based mid-level metric that delivers fine-grained prompt-image alignment and shows strong correlation with human judgments; and (3) high-level metrics for aesthetics and human preference. Using MMIG-Bench, we benchmark 17 state-of-the-art models, including Gemini 2.5 Pro, FLUX, DreamBooth, and IP-Adapter, and validate our metrics with 32k human ratings, yielding in-depth insights into architecture and data design.