MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation

TL;DR

MultiBanana is a comprehensive benchmark designed to evaluate and challenge multi-reference text-to-image generation models across diverse, complex scenarios including multiple references, domain and scale mismatches, rare concepts, and multilingual inputs.

Contribution

The paper introduces MultiBanana, a new benchmark dataset that covers diverse and challenging multi-reference scenarios to better evaluate and compare text-to-image models.

Findings

Models show varied performance across different challenges.

Common failure modes include domain mismatch and rare concept handling.

Benchmark reveals areas for future model improvements.

Abstract

Recent text-to-image generation models have acquired the ability of multi-reference generation and editing; that is, to inherit the appearance of subjects from multiple reference images and re-render them in new contexts. However, existing benchmark datasets often focus on generation using a single or a few reference images, which prevents us from measuring progress in model performance or identifying weaknesses when following instructions with a larger number of references. In addition, their task definitions are still vague, limited to axes such as ``what to edit'' or ``how many references are given'', and therefore fail to capture the challenges inherent in combining heterogeneous references. To address this gap, we introduce MultiBanana, which is designed to assess the edge of model capabilities by widely covering problems specific to multi-reference settings: (1) varying the number of references (up to 8), (2) domain mismatch among references (e.g., photo vs. anime), (3) scale mismatch between reference and target scenes, (4) references containing rare concepts (e.g., a red banana), and (5) multilingual textual references for rendering. Our analysis among a variety of text-to-image models reveals their respective performances, typical failure modes, and areas for improvement. MultiBanana is released as an open benchmark to push the boundaries and establish a standardized basis for fair comparison in multi-reference image generation. Our data and code are available at https://github.com/matsuolab/multibanana .