CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Models

TL;DR

CLIPAway is a novel method that uses CLIP embeddings to improve object removal in diffusion-based inpainting, achieving more accurate and seamless results without extensive training or manual annotations.

Contribution

It introduces a flexible, plug-and-play approach leveraging CLIP embeddings to focus on background regions, enhancing diffusion model inpainting for object removal.

Findings

Improves inpainting accuracy and quality for object removal

Reduces hallucinations of removed objects

Compatible with various diffusion-based techniques

Abstract

Advanced image editing techniques, particularly inpainting, are essential for seamlessly removing unwanted elements while preserving visual integrity. Traditional GAN-based methods have achieved notable success, but recent advancements in diffusion models have produced superior results due to their training on large-scale datasets, enabling the generation of remarkably realistic inpainted images. Despite their strengths, diffusion models often struggle with object removal tasks without explicit guidance, leading to unintended hallucinations of the removed object. To address this issue, we introduce CLIPAway, a novel approach leveraging CLIP embeddings to focus on background regions while excluding foreground elements. CLIPAway enhances inpainting accuracy and quality by identifying embeddings that prioritize the background, thus achieving seamless object removal. Unlike other methods that rely on specialized training datasets or costly manual annotations, CLIPAway provides a flexible, plug-and-play solution compatible with various diffusion-based inpainting techniques.