Pinco: Position-induced Consistent Adapter for Diffusion Transformer in Foreground-conditioned Inpainting

TL;DR

Pinco introduces a novel adapter for diffusion transformers that improves foreground-conditioned inpainting by enhancing shape preservation, text alignment, and feature extraction, leading to superior image quality.

Contribution

The paper proposes a plug-and-play adapter with a self-consistent attention mechanism, decoupled feature extraction, and shared positional embeddings for improved foreground inpainting.

Findings

Achieves better shape preservation of foreground subjects.

Enhances alignment between generated background and text descriptions.

Improves training efficiency and overall inpainting quality.

Abstract

Foreground-conditioned inpainting aims to seamlessly fill the background region of an image by utilizing the provided foreground subject and a text description. While existing T2I-based image inpainting methods can be applied to this task, they suffer from issues of subject shape expansion, distortion, or impaired ability to align with the text description, resulting in inconsistencies between the visual elements and the text description. To address these challenges, we propose Pinco, a plug-and-play foreground-conditioned inpainting adapter that generates high-quality backgrounds with good text alignment while effectively preserving the shape of the foreground subject. Firstly, we design a Self-Consistent Adapter that integrates the foreground subject features into the layout-related self-attention layer, which helps to alleviate conflicts between the text and subject features by ensuring that the model can effectively consider the foreground subject's characteristics while processing the overall image layout. Secondly, we design a Decoupled Image Feature Extraction method that employs distinct architectures to extract semantic and spatial features separately, significantly improving subject feature extraction and ensuring high-quality preservation of the subject's shape. Thirdly, to ensure precise utilization of the extracted features and to focus attention on the subject region, we introduce a Shared Positional Embedding Anchor, greatly improving the model's understanding of subject features and boosting training efficiency. Extensive experiments demonstrate that our method achieves superior performance and efficiency in foreground-conditioned inpainting.