N\"UWA-LIP: Language Guided Image Inpainting with Defect-free VQGAN

TL;DR

NÜWA-LIP introduces a novel language-guided image inpainting method that leverages defect-free VQGAN and multi-perspective sequence modeling to improve visual quality and robustness, outperforming recent baselines.

Contribution

The paper proposes NÜWA-LIP, combining defect-free VQGAN with multi-perspective sequence modeling to address receptive spreading and information loss in language-guided image inpainting.

Findings

DF-VQGAN is more robust than VQGAN.

NÜWA-LIP outperforms recent baselines on open-domain benchmarks.

The method effectively preserves non-defective regions while filling in defective areas.

Abstract

Language guided image inpainting aims to fill in the defective regions of an image under the guidance of text while keeping non-defective regions unchanged. However, the encoding process of existing models suffers from either receptive spreading of defective regions or information loss of non-defective regions, giving rise to visually unappealing inpainting results. To address the above issues, this paper proposes N\"UWA-LIP by incorporating defect-free VQGAN (DF-VQGAN) with multi-perspective sequence to sequence (MP-S2S). In particular, DF-VQGAN introduces relative estimation to control receptive spreading and adopts symmetrical connections to protect information. MP-S2S further enhances visual information from complementary perspectives, including both low-level pixels and high-level tokens. Experiments show that DF-VQGAN performs more robustness than VQGAN. To evaluate the inpainting performance of our model, we built up 3 open-domain benchmarks, where N\"UWA-LIP is also superior to recent strong baselines.