Stereoscopic video deblurring transformer

TL;DR

This paper introduces a new Transformer-based framework for deblurring stereoscopic videos, leveraging attention mechanisms and stereo-viewpoint information to achieve state-of-the-art results.

Contribution

The paper proposes a novel Transformer-based framework with a parallax attention module for efficient stereo video deblurring.

Findings

The proposed method outperforms existing image and video deblurring techniques on stereo video datasets.

Ablation studies confirm the effectiveness of the self-attention and parallax attention modules in improving deblurring performance.

Abstract

Stereoscopic cameras, such as those in mobile phones and various recent intelligent systems, are becoming increasingly common. Multiple variables can impact the stereo video quality, e.g., blur distortion due to camera/object movement. Monocular image/video deblurring is a mature research field, while there is limited research on stereoscopic content deblurring. This paper introduces a new Transformer-based stereo video deblurring framework with two crucial new parts: a self-attention layer and a feed-forward layer that realizes and aligns the correlation among various video frames. The traditional fully connected (FC) self-attention layer fails to utilize data locality effectively, as it depends on linear layers for calculating attention maps The Vision Transformer, on the other hand, also has this limitation, as it takes image patches as inputs to model global spatial information. 3D convolutional neural networks (3D CNNs) process successive frames to correct motion blur in the stereo video. Besides, our method uses other stereo-viewpoint information to assist deblurring. The parallax attention module (PAM) is significantly improved to combine the stereo and cross-view information for more deblurring. An extensive ablation study validates that our method efficiently deblurs the stereo videos based on the experiments on two publicly available stereo video datasets. Experimental results of our approach demonstrate state-of-the-art performance compared to the image and video deblurring techniques by a large margin.

Figures