Deep Video Restoration for Under-Display Camera

TL;DR

This paper introduces the first large-scale dataset and a transformer-based method for video restoration of Under-Display Camera images, addressing severe degradation issues and setting new performance benchmarks.

Contribution

It creates the first UDC video restoration dataset, PexelsUDC, and proposes a novel transformer-based model that effectively exploits spatial and temporal information for improved restoration.

Findings

Our method achieves state-of-the-art results on PexelsUDC.

Existing video restoration methods have limitations on UDC videos.

The dataset and benchmark facilitate future research in UDC video restoration.

Abstract

Images or videos captured by the Under-Display Camera (UDC) suffer from severe degradation, such as saturation degeneration and color shift. While restoration for UDC has been a critical task, existing works of UDC restoration focus only on images. UDC video restoration (UDC-VR) has not been explored in the community. In this work, we first propose a GAN-based generation pipeline to simulate the realistic UDC degradation process. With the pipeline, we build the first large-scale UDC video restoration dataset called PexelsUDC, which includes two subsets named PexelsUDC-T and PexelsUDC-P corresponding to different displays for UDC. Using the proposed dataset, we conduct extensive benchmark studies on existing video restoration methods and observe their limitations on the UDC-VR task. To this end, we propose a novel transformer-based baseline method that adaptively enhances degraded videos. The key components of the method are a spatial branch with local-aware transformers, a temporal branch embedded temporal transformers, and a spatial-temporal fusion module. These components drive the model to fully exploit spatial and temporal information for UDC-VR. Extensive experiments show that our method achieves state-of-the-art performance on PexelsUDC. The benchmark and the baseline method are expected to promote the progress of UDC-VR in the community, which will be made public.