Aligned Stable Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency

TL;DR

This paper introduces ASUKA, a framework that enhances generative image inpainting by reducing unwanted object hallucination and color inconsistency through reconstruction priors and a specialized VAE decoder, improving realism and color fidelity.

Contribution

The paper proposes a novel post-hoc framework, ASUKA, that significantly mitigates hallucinated objects and color shifts in pre-trained inpainting models without retraining them.

Findings

ASUKA suppresses unwanted object hallucination effectively.

ASUKA improves color consistency in inpainted images.

Outperforms existing inpainting methods on benchmark datasets.

Abstract

Generative image inpainting can produce realistic, high-fidelity results even with large, irregular masks. However, existing methods still face key issues that make inpainted images look unnatural. In this paper, we identify two main problems: (1) Unwanted object insertion: generative models may hallucinate arbitrary objects in the masked region that do not match the surrounding context. (2) Color inconsistency: inpainted regions often exhibit noticeable color shifts, leading to smeared textures and degraded image quality. We analyze the underlying causes of these issues and propose efficient post-hoc solutions for pre-trained inpainting models. Specifically, we introduce the principled framework of Aligned Stable inpainting with UnKnown Areas prior (ASUKA). To reduce unwanted object insertion, we use reconstruction-based priors to guide the generative model, suppressing hallucinated objects while preserving generative flexibility. To address color inconsistency, we design a specialized VAE decoder that formulates latent-to-image decoding as a local harmonization task. This design significantly reduces color shifts and produces more color-consistent results. We implement ASUKA on two representative inpainting architectures: a U-Net-based model and a DiT-based model. We analyze and propose lightweight injection strategies that minimize interference with the model's original generation capacity while ensuring the mitigation of the two issues. We evaluate ASUKA using the Places2 dataset and MISATO, our proposed diverse benchmark. Experiments show that ASUKA effectively suppresses object hallucination and improves color consistency, outperforming standard diffusion, rectified flow models, and other inpainting methods. Dataset, models and codes will be released in github.