WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing

TL;DR

WiseEdit is a comprehensive benchmark designed to evaluate advanced image editing models on cognition- and creativity-informed tasks, addressing limitations of existing benchmarks by incorporating deep task depth, broad knowledge types, and complex challenges.

Contribution

It introduces WiseEdit, a knowledge-intensive benchmark with a novel three-step task decomposition and diverse knowledge types, enabling holistic assessment of image editing models.

Findings

Reveals limitations of current models in cognitive reasoning and creative composition.

Provides a large dataset of 1,220 test cases for thorough evaluation.

Establishes a new standard for holistic benchmarking in image editing.

Abstract

Recent image editing models boast next-level intelligent capabilities, facilitating cognition- and creativity-informed image editing. Yet, existing benchmarks provide too narrow a scope for evaluation, failing to holistically assess these advanced abilities. To address this, we introduce WiseEdit, a knowledge-intensive benchmark for comprehensive evaluation of cognition- and creativity-informed image editing, featuring deep task depth and broad knowledge breadth. Drawing an analogy to human cognitive creation, WiseEdit decomposes image editing into three cascaded steps, i.e., Awareness, Interpretation, and Imagination, each corresponding to a task that poses a challenge for models to complete at the specific step. It also encompasses complex tasks, where none of the three steps can be finished easily. Furthermore, WiseEdit incorporates three fundamental types of knowledge: Declarative, Procedural, and Metacognitive knowledge. Ultimately, WiseEdit comprises 1,220 test cases, objectively revealing the limitations of SoTA image editing models in knowledge-based cognitive reasoning and creative composition capabilities. The benchmark, evaluation code, and the generated images of each model will be made publicly available soon. Project Page: https://qnancy.github.io/wiseedit_project_page/.