PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control

TL;DR

PainterNet is a flexible plugin for diffusion models that improves image inpainting by enhancing local focus and user control, leading to more consistent and high-quality results.

Contribution

It introduces local prompt input, Attention Control Points, and Actual-Token Attention Loss to better align inpainted content with user prompts and habits.

Findings

Outperforms state-of-the-art models in image quality

Achieves higher global and local text consistency

Demonstrates effective user habit simulation in inpainting

Abstract

Recently, diffusion models have exhibited superior performance in the area of image inpainting. Inpainting methods based on diffusion models can usually generate realistic, high-quality image content for masked areas. However, due to the limitations of diffusion models, existing methods typically encounter problems in terms of semantic consistency between images and text, and the editing habits of users. To address these issues, we present PainterNet, a plugin that can be flexibly embedded into various diffusion models. To generate image content in the masked areas that highly aligns with the user input prompt, we proposed local prompt input, Attention Control Points (ACP), and Actual-Token Attention Loss (ATAL) to enhance the model's focus on local areas. Additionally, we redesigned the MASK generation algorithm in training and testing dataset to simulate the user's habit of applying MASK, and introduced a customized new training dataset, PainterData, and a benchmark dataset, PainterBench. Our extensive experimental analysis exhibits that PainterNet surpasses existing state-of-the-art models in key metrics including image quality and global/local text consistency.