EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Images

TL;DR

EraseDraw introduces a novel approach to image editing by learning object insertion through erasing objects from images, leveraging high-quality data generated from object removal to improve spatially and visually consistent object insertion in diverse images.

Contribution

The paper presents a scalable data generation pipeline for training object insertion models by reversing object removal, leading to a state-of-the-art diffusion model for in-the-wild image editing.

Findings

Achieves state-of-the-art results in object insertion tasks.

Performs well on diverse prompts and images across various domains.

Automates iterative insertion using CLIP-guided beam search.

Abstract

Creative processes such as painting often involve creating different components of an image one by one. Can we build a computational model to perform this task? Prior works often fail by making global changes to the image, inserting objects in unrealistic spatial locations, and generating inaccurate lighting details. We observe that while state-of-the-art models perform poorly on object insertion, they can remove objects and erase the background in natural images very well. Inverting the direction of object removal, we obtain high-quality data for learning to insert objects that are spatially, physically, and optically consistent with the surroundings. With this scalable automatic data generation pipeline, we can create a dataset for learning object insertion, which is used to train our proposed text conditioned diffusion model. Qualitative and quantitative experiments have shown that our model achieves state-of-the-art results in object insertion, particularly for in-the-wild images. We show compelling results on diverse insertion prompts and images across various domains.In addition, we automate iterative insertion by combining our insertion model with beam search guided by CLIP.