Blur Interpolation Transformer for Real-World Motion from Blur

TL;DR

This paper introduces a novel transformer-based model called BiT for recovering motion from blurred videos, significantly improving visual quality and generalization to real-world data through innovative training strategies and a new real-world dataset.

Contribution

We propose a blur interpolation transformer (BiT) with dual-end supervision and symmetric ensembling, and create the first real-world blur-sharp video dataset for better generalization.

Findings

BiT outperforms state-of-the-art on Adobe240 dataset.

The real-world dataset improves model generalization.

BiT effectively captures temporal correlations in blur.

Abstract

This paper studies the challenging problem of recovering motion from blur, also known as joint deblurring and interpolation or blur temporal super-resolution. The challenges are twofold: 1) the current methods still leave considerable room for improvement in terms of visual quality even on the synthetic dataset, and 2) poor generalization to real-world data. To this end, we propose a blur interpolation transformer (BiT) to effectively unravel the underlying temporal correlation encoded in blur. Based on multi-scale residual Swin transformer blocks, we introduce dual-end temporal supervision and temporally symmetric ensembling strategies to generate effective features for time-varying motion rendering. In addition, we design a hybrid camera system to collect the first real-world dataset of one-to-many blur-sharp video pairs. Experimental results show that BiT has a significant gain over the state-of-the-art methods on the public dataset Adobe240. Besides, the proposed real-world dataset effectively helps the model generalize well to real blurry scenarios. Code and data are available at https://github.com/zzh-tech/BiT.