DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models

TL;DR

DreamInpainter is a novel method that combines text prompts and exemplar images to improve subject-driven image inpainting, balancing fidelity and editability through a two-step process and regularization techniques.

Contribution

It introduces a new task of text-guided subject-driven image inpainting and proposes a two-step approach with regularization to enhance performance.

Findings

Outperforms existing methods in visual quality and identity preservation.

Effectively balances subject fidelity and editability with the proposed approach.

Demonstrates superior control using text prompts in inpainting tasks.

Abstract

This study introduces Text-Guided Subject-Driven Image Inpainting, a novel task that combines text and exemplar images for image inpainting. While both text and exemplar images have been used independently in previous efforts, their combined utilization remains unexplored. Simultaneously accommodating both conditions poses a significant challenge due to the inherent balance required between editability and subject fidelity. To tackle this challenge, we propose a two-step approach DreamInpainter. First, we compute dense subject features to ensure accurate subject replication. Then, we employ a discriminative token selection module to eliminate redundant subject details, preserving the subject's identity while allowing changes according to other conditions such as mask shape and text prompts. Additionally, we introduce a decoupling regularization technique to enhance text control in the presence of exemplar images. Our extensive experiments demonstrate the superior performance of our method in terms of visual quality, identity preservation, and text control, showcasing its effectiveness in the context of text-guided subject-driven image inpainting.