Towards Robust Sequential Decomposition for Complex Image Editing

TL;DR

This paper investigates robust sequential decomposition for complex image editing, balancing benefits and errors of different paradigms, and demonstrates improved performance through synthetic data training and transfer to real images.

Contribution

It introduces a synthetic data pipeline for decomposing complex editing tasks and shows that proper sequential paradigms enhance robustness and generalization in image editing.

Findings

Sequential decomposition improves robustness with increasing task complexity.

Synthetic data training enhances editing performance and generalization.

Transfer learning from synthetic to real images is effective.

Abstract

Recent advances in visual generative models have enabled high-fidelity image editing guided by human instructions. However, these models often struggle with complex instructions involving combinatorial editing operations or inter-step dependencies. This difficulty stems from the limitations of two canonical paradigms: (1) single-turn editing, which attempts to apply all instructed edits in one pass, often fails to parse the complex instruction accurately and causes undesired edits; and (2) sequential editing can decompose the task into simpler steps but suffers from compounding errors introduced by the sequential execution, leading to low-fidelity results. To derive a robust solution for complex image editing, we examine editing behaviors of different paradigms under a unified in-context editing framework, and study how the benefits of sequential decomposition can be balanced against its error-accumulation drawbacks. We further develop a synthetic data pipeline that constructs editing tasks of varying instruction complexity, allowing us to curate a large-scale editing dataset with high-quality decomposed sequences. By finetuning on synthetic data, we discovered that with properly designed editing paradigms, sequential decomposition yields robust improvements even as task complexity increases. Furthermore, the decomposition skills learned from synthetic tasks can transfer to real images by co-training with real-world editing data, demonstrating the promise of sim-to-real generalization for tackling complex image editing across broader domains.