Image Editing As Programs with Diffusion Models

TL;DR

This paper introduces IEAP, a novel framework using diffusion transformers and programmatic decomposition to improve instruction-driven image editing, especially for complex and structural changes.

Contribution

The paper presents a unified, modular image editing framework that decomposes complex instructions into atomic operations, enhancing robustness and generalization in diffusion-based editing models.

Findings

IEAP outperforms state-of-the-art methods on standard benchmarks.

The modular approach improves handling of complex, multi-step edits.

IEAP achieves higher accuracy and semantic fidelity in image editing.

Abstract

While diffusion models have achieved remarkable success in text-to-image generation, they encounter significant challenges with instruction-driven image editing. Our research highlights a key challenge: these models particularly struggle with structurally inconsistent edits that involve substantial layout changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture. At its core, IEAP approaches instructional editing through a reductionist lens, decomposing complex editing instructions into sequences of atomic operations. Each operation is implemented via a lightweight adapter sharing the same DiT backbone and is specialized for a specific type of edit. Programmed by a vision-language model (VLM)-based agent, these operations collaboratively support arbitrary and structurally inconsistent transformations. By modularizing and sequencing edits in this way, IEAP generalizes robustly across a wide range of editing tasks, from simple adjustments to substantial structural changes. Extensive experiments demonstrate that IEAP significantly outperforms state-of-the-art methods on standard benchmarks across various editing scenarios. In these evaluations, our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions. Codes are available at https://github.com/YujiaHu1109/IEAP.