MagicEraser: Erasing Any Objects via Semantics-Aware Control

TL;DR

MagicEraser is a diffusion model-based framework that effectively erases objects and generates harmonious backgrounds with controllable content, addressing limitations of previous GAN and diffusion methods.

Contribution

It introduces a novel two-phase framework with prompt tuning and semantics-aware attention refocus modules for improved object erasure and background generation.

Findings

Outperforms previous methods in object erasure quality

Produces more harmonious and artifact-free backgrounds

Demonstrates effective control over generated content

Abstract

The traditional image inpainting task aims to restore corrupted regions by referencing surrounding background and foreground. However, the object erasure task, which is in increasing demand, aims to erase objects and generate harmonious background. Previous GAN-based inpainting methods struggle with intricate texture generation. Emerging diffusion model-based algorithms, such as Stable Diffusion Inpainting, exhibit the capability to generate novel content, but they often produce incongruent results at the locations of the erased objects and require high-quality text prompt inputs. To address these challenges, we introduce MagicEraser, a diffusion model-based framework tailored for the object erasure task. It consists of two phases: content initialization and controllable generation. In the latter phase, we develop two plug-and-play modules called prompt tuning and semantics-aware attention refocus. Additionally, we propose a data construction strategy that generates training data specially suitable for this task. MagicEraser achieves fine and effective control of content generation while mitigating undesired artifacts. Experimental results highlight a valuable advancement of our approach in the object erasure task.