Integrating Spatial and Frequency Information for Under-Display Camera Image Restoration

TL;DR

This paper introduces SFIM, a novel neural network architecture that integrates spatial and frequency domain information to effectively restore images captured by under-display cameras, addressing complex degradations.

Contribution

The paper proposes a multi-level DNN architecture combining CNNs and FFT-based models for joint spatial-frequency image restoration in UDCs, a novel approach in this context.

Findings

SFIM outperforms state-of-the-art methods on three UDC benchmarks.

Effective integration of local and global information improves restoration quality.

The architecture demonstrates superior handling of noise, blur, and flare artifacts.

Abstract

Under-Display Camera (UDC) houses a digital camera lens under a display panel. However, UDC introduces complex degradations such as noise, blur, decrease in transmittance, and flare. Despite the remarkable progress, previous research on UDC mainly focuses on eliminating diffraction in the spatial domain and rarely explores its potential in the frequency domain. It is essential to consider both the spatial and frequency domains effectively. For example, degradations, such as noise and blur, can be addressed by local information (e.g., CNN kernels in the spatial domain). At the same time, tackling flares may require leveraging global information (e.g., the frequency domain). In this paper, we revisit the UDC degradations in the Fourier space and figure out intrinsic frequency priors that imply the presence of the flares. Based on this observation, we propose a novel multi-level DNN architecture called SFIM. It efficiently restores UDC-distorted images by integrating local and global (the collective contribution of all points in the image) information. The architecture exploits CNNs to capture local information and FFT-based models to capture global information. SFIM comprises a spatial domain block (SDB), a Frequency Domain Block (FDB), and an Attention-based Multi-level Integration Block (AMIB). Specifically, SDB focuses more on detailed textures such as noise and blur, FDB emphasizes irregular texture loss in extensive areas such as flare, and AMIB enables effective cross-domain interaction. SFIM's superior performance over state-of-the-art approaches is demonstrated through rigorous quantitative and qualitative assessments across three UDC benchmarks.