IDEA-Bench: How Far are Generative Models from Professional Designing?

TL;DR

IDEA-Bench is a new comprehensive benchmark designed to evaluate the capabilities of generative models in complex professional design tasks, revealing significant gaps in current model performance and guiding future improvements.

Contribution

The paper introduces IDEA-Bench, a large-scale benchmark with 100 real-world design tasks and 275 test cases, along with evaluation tools and a leaderboard to advance generative model development for design applications.

Findings

Current models achieve only 22.48 and 6.81 scores on IDEA-Bench.

Significant challenges remain in applying generative models to professional design tasks.

Benchmark and evaluation tools are released to foster further research.

Abstract

Real-world design tasks - such as picture book creation, film storyboard development using character sets, photo retouching, visual effects, and font transfer - are highly diverse and complex, requiring deep interpretation and extraction of various elements from instructions, descriptions, and reference images. The resulting images often implicitly capture key features from references or user inputs, making it challenging to develop models that can effectively address such varied tasks. While existing visual generative models can produce high-quality images based on prompts, they face significant limitations in professional design scenarios that involve varied forms and multiple inputs and outputs, even when enhanced with adapters like ControlNets and LoRAs. To address this, we introduce IDEA-Bench, a comprehensive benchmark encompassing 100 real-world design tasks, including rendering, visual effects, storyboarding, picture books, fonts, style-based, and identity-preserving generation, with 275 test cases to thoroughly evaluate a model's general-purpose generation capabilities. Notably, even the best-performing model only achieves 22.48 on IDEA-Bench, while the best general-purpose model only achieves 6.81. We provide a detailed analysis of these results, highlighting the inherent challenges and providing actionable directions for improvement. Additionally, we provide a subset of 18 representative tasks equipped with multimodal large language model (MLLM)-based auto-evaluation techniques to facilitate rapid model development and comparison. We releases the benchmark data, evaluation toolkits, and an online leaderboard at https://github.com/ali-vilab/IDEA-Bench, aiming to drive the advancement of generative models toward more versatile and applicable intelligent design systems.