Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration

TL;DR

This paper introduces SGSFormer, a segmentation-guided sparse Transformer that improves under-display camera image restoration by focusing attention on relevant regions and reducing noise, outperforming traditional dense attention methods.

Contribution

The paper proposes a novel segmentation-guided sparse Transformer architecture that leverages instance segmentation maps to enhance UDC image restoration by filtering redundant information.

Findings

Sparse self-attention reduces noise and redundancy.

Guided attention improves restoration quality.

Outperforms dense Transformer methods in experiments.

Abstract

Under-Display Camera (UDC) is an emerging technology that achieves full-screen display via hiding the camera under the display panel. However, the current implementation of UDC causes serious degradation. The incident light required for camera imaging undergoes attenuation and diffraction when passing through the display panel, leading to various artifacts in UDC imaging. Presently, the prevailing UDC image restoration methods predominantly utilize convolutional neural network architectures, whereas Transformer-based methods have exhibited superior performance in the majority of image restoration tasks. This is attributed to the Transformer's capability to sample global features for the local reconstruction of images, thereby achieving high-quality image restoration. In this paper, we observe that when using the Vision Transformer for UDC degraded image restoration, the global attention samples a large amount of redundant information and noise. Furthermore, compared to the ordinary Transformer employing dense attention, the Transformer utilizing sparse attention can alleviate the adverse impact of redundant information and noise. Building upon this discovery, we propose a Segmentation Guided Sparse Transformer method (SGSFormer) for the task of restoring high-quality images from UDC degraded images. Specifically, we utilize sparse self-attention to filter out redundant information and noise, directing the model's attention to focus on the features more relevant to the degraded regions in need of reconstruction. Moreover, we integrate the instance segmentation map as prior information to guide the sparse self-attention in filtering and focusing on the correct regions.