You Only Align Once: Bidirectional Interaction for Spatial-Temporal Video Super-Resolution

TL;DR

This paper introduces a bidirectional interaction network for spatial-temporal video super-resolution that reduces redundant alignments, improves information flow, and enhances efficiency, outperforming existing methods.

Contribution

It proposes a novel recurrent network with only one alignment and fusion step, utilizing bidirectional inference and a Hybrid Fusion Module for superior ST-VSR performance.

Findings

Outperforms state-of-the-art methods in accuracy.

Reduces computational cost by approximately 22%.

Effectively exploits bidirectional motion and spatial information.

Abstract

Spatial-Temporal Video Super-Resolution (ST-VSR) technology generates high-quality videos with higher resolution and higher frame rates. Existing advanced methods accomplish ST-VSR tasks through the association of Spatial and Temporal video super-resolution (S-VSR and T-VSR). These methods require two alignments and fusions in S-VSR and T-VSR, which is obviously redundant and fails to sufficiently explore the information flow of consecutive spatial LR frames. Although bidirectional learning (future-to-past and past-to-future) was introduced to cover all input frames, the direct fusion of final predictions fails to sufficiently exploit intrinsic correlations of bidirectional motion learning and spatial information from all frames. We propose an effective yet efficient recurrent network with bidirectional interaction for ST-VSR, where only one alignment and fusion is needed. Specifically, it first performs backward inference from future to past, and then follows forward inference to super-resolve intermediate frames. The backward and forward inferences are assigned to learn structures and details to simplify the learning task with joint optimizations. Furthermore, a Hybrid Fusion Module (HFM) is designed to aggregate and distill information to refine spatial information and reconstruct high-quality video frames. Extensive experiments on two public datasets demonstrate that our method outperforms state-of-the-art methods in efficiency, and reduces calculation cost by about 22%.