Lightweight Image Inpainting by Stripe Window Transformer with Joint Attention to CNN

TL;DR

This paper introduces a lightweight image inpainting model that combines a stripe window transformer with CNNs and a new color-focused loss function, achieving improved texture and color detail reconstruction on standard datasets.

Contribution

The paper presents a novel lightweight inpainting model integrating a stripe window transformer with CNNs and a new HSV-based loss function for enhanced color detail.

Findings

Outperforms state-of-the-art methods on Places2 and CelebA datasets.

Effectively reconstructs textures and structures with improved color fidelity.

Demonstrates the efficiency of the lightweight model in practical scenarios.

Abstract

Image inpainting is an important task in computer vision. As admirable methods are presented, the inpainted image is getting closer to reality. However, the result is still not good enough in the reconstructed texture and structure based on human vision. Although recent advances in computer hardware have enabled the development of larger and more complex models, there is still a need for lightweight models that can be used by individuals and small-sized institutions. Therefore, we propose a lightweight model that combines a specialized transformer with a traditional convolutional neural network (CNN). Furthermore, we have noticed most researchers only consider three primary colors (RGB) in inpainted images, but we think this is not enough. So we propose a new loss function to intensify color details. Extensive experiments on commonly seen datasets (Places2 and CelebA) validate the efficacy of our proposed model compared with other state-of-the-art methods. Index Terms: HSV color space, image inpainting, joint attention, stripe window, transformer