High-Fidelity Pluralistic Image Completion with Transformers

TL;DR

This paper introduces a hybrid approach combining transformers and CNNs for high-fidelity, diverse, and globally coherent image completion, outperforming existing methods especially on high-resolution images.

Contribution

It presents a novel hybrid model that leverages transformers for global structure and CNNs for local detail, advancing pluralistic image completion.

Findings

Significant improvement in image fidelity over state-of-the-art methods.

Enhanced diversity and realism in completed images.

Strong generalization to large masks and diverse datasets like ImageNet.

Abstract

Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. The proposed method vastly outperforms state-of-the-art methods in terms of three aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet.