MultiRef: Controllable Image Generation with Multiple Visual References

TL;DR

This paper introduces MultiRef, a new benchmark and dataset for evaluating controllable image generation using multiple visual references, revealing current models' limitations and guiding future improvements.

Contribution

The paper presents MultiRef-bench and MultiRef datasets, along with an analysis of state-of-the-art models' performance on multi-reference image generation tasks.

Findings

State-of-the-art models achieve only 66.6% accuracy on synthetic samples.

Models perform better on real-world samples, reaching 79.0%.

Current systems struggle with multi-reference conditioning, indicating room for improvement.

Abstract

Visual designers naturally draw inspiration from multiple visual references, combining diverse elements and aesthetic principles to create artwork. However, current image generative frameworks predominantly rely on single-source inputs -- either text prompts or individual reference images. In this paper, we focus on the task of controllable image generation using multiple visual references. We introduce MultiRef-bench, a rigorous evaluation framework comprising 990 synthetic and 1,000 real-world samples that require incorporating visual content from multiple reference images. The synthetic samples are synthetically generated through our data engine RefBlend, with 10 reference types and 33 reference combinations. Based on RefBlend, we further construct a dataset MultiRef containing 38k high-quality images to facilitate further research. Our experiments across three interleaved image-text models (i.e., OmniGen, ACE, and Show-o) and six agentic frameworks (e.g., ChatDiT and LLM + SD) reveal that even state-of-the-art systems struggle with multi-reference conditioning, with the best model OmniGen achieving only 66.6% in synthetic samples and 79.0% in real-world cases on average compared to the golden answer. These findings provide valuable directions for developing more flexible and human-like creative tools that can effectively integrate multiple sources of visual inspiration. The dataset is publicly available at: https://multiref.github.io/.