A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting

TL;DR

PowerPaint is a versatile image inpainting model that uses learnable task prompts and fine-tuning strategies to excel in multiple inpainting tasks, including background filling, object synthesis, and shape-guided inpainting, achieving state-of-the-art results.

Contribution

The paper introduces learnable task prompts and tailored fine-tuning strategies, enabling a single model to perform diverse inpainting tasks with high quality.

Findings

Achieves state-of-the-art performance in multiple inpainting tasks.

Effectively uses task prompts for object removal and shape-guided inpainting.

Demonstrates versatility and controllability in image inpainting applications.

Abstract

Advancing image inpainting is challenging as it requires filling user-specified regions for various intents, such as background filling and object synthesis. Existing approaches focus on either context-aware filling or object synthesis using text descriptions. However, achieving both tasks simultaneously is challenging due to differing training strategies. To overcome this challenge, we introduce PowerPaint, the first high-quality and versatile inpainting model that excels in multiple inpainting tasks. First, we introduce learnable task prompts along with tailored fine-tuning strategies to guide the model's focus on different inpainting targets explicitly. This enables PowerPaint to accomplish various inpainting tasks by utilizing different task prompts, resulting in state-of-the-art performance. Second, we demonstrate the versatility of the task prompt in PowerPaint by showcasing its effectiveness as a negative prompt for object removal. Moreover, we leverage prompt interpolation techniques to enable controllable shape-guided object inpainting, enhancing the model's applicability in shape-guided applications. Finally, we conduct extensive experiments and applications to verify the effectiveness of PowerPaint. We release our codes and models on our project page: https://powerpaint.github.io/.