ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies

TL;DR

This paper introduces ComplexBench-Edit, a benchmark for evaluating complex, multi-step image editing instructions, along with a new consistency metric and a Chain-of-Thought approach that improves model performance on such tasks.

Contribution

It presents a novel benchmark, a new evaluation method, and a Chain-of-Thought technique to better assess and enhance models' ability to handle complex image editing instructions.

Findings

ComplexBench-Edit effectively differentiates model capabilities.

The CoT-based approach significantly improves handling of complex instructions.

New consistency metric accurately evaluates non-modified regions.

Abstract

Text-driven image editing has achieved remarkable success in following single instructions. However, real-world scenarios often involve complex, multi-step instructions, particularly ``chain'' instructions where operations are interdependent. Current models struggle with these intricate directives, and existing benchmarks inadequately evaluate such capabilities. Specifically, they often overlook multi-instruction and chain-instruction complexities, and common consistency metrics are flawed. To address this, we introduce ComplexBench-Edit, a novel benchmark designed to systematically assess model performance on complex, multi-instruction, and chain-dependent image editing tasks. ComplexBench-Edit also features a new vision consistency evaluation method that accurately assesses non-modified regions by excluding edited areas. Furthermore, we propose a simple yet powerful Chain-of-Thought (CoT)-based approach that significantly enhances the ability of existing models to follow complex instructions. Our extensive experiments demonstrate ComplexBench-Edit's efficacy in differentiating model capabilities and highlight the superior performance of our CoT-based method in handling complex edits. The data and code are released at https://github.com/llllly26/ComplexBench-Edit.