Editable Image Elements for Controllable Synthesis

TL;DR

This paper introduces a novel image representation called 'image elements' that enables intuitive spatial editing of images using diffusion models, facilitating tasks like resizing, rearrangement, and removal with high fidelity.

Contribution

We propose a new encoding method that allows effective spatial editing of images within diffusion models, addressing the challenge of image inversion and editing.

Findings

Effective for various editing tasks like object resizing and removal

Produces realistic edited images with high fidelity

Enables intuitive user-driven image manipulation

Abstract

Diffusion models have made significant advances in text-guided synthesis tasks. However, editing user-provided images remains challenging, as the high dimensional noise input space of diffusion models is not naturally suited for image inversion or spatial editing. In this work, we propose an image representation that promotes spatial editing of input images using a diffusion model. Concretely, we learn to encode an input into "image elements" that can faithfully reconstruct an input image. These elements can be intuitively edited by a user, and are decoded by a diffusion model into realistic images. We show the effectiveness of our representation on various image editing tasks, such as object resizing, rearrangement, dragging, de-occlusion, removal, variation, and image composition. Project page: https://jitengmu.github.io/Editable_Image_Elements/