Optical Flow Reusing for High-Efficiency Space-Time Video Super Resolution

TL;DR

This paper introduces OFR-BRN, a novel bidirectional recurrent network that reuses optical flow to enhance space-time video super-resolution, significantly reducing computational costs while improving accuracy in reconstructing high-resolution, high-frame-rate videos.

Contribution

The paper proposes an optical-flow-reuse strategy within a bidirectional recurrent network to efficiently utilize long-range temporal information for space-time video super-resolution.

Findings

Outperforms state-of-the-art methods in accuracy.

Reduces computational burden compared to traditional methods.

Effectively restores details using intermediate flow estimation.

Abstract

In this paper, we consider the task of space-time video super-resolution (ST-VSR), which can increase the spatial resolution and frame rate for a given video simultaneously. Despite the remarkable progress of recent methods, most of them still suffer from high computational costs and inefficient long-range information usage. To alleviate these problems, we propose a Bidirectional Recurrence Network (BRN) with the optical-flow-reuse strategy to better use temporal knowledge from long-range neighboring frames for high-efficiency reconstruction. Specifically, an efficient and memory-saving multi-frame motion utilization strategy is proposed by reusing the intermediate flow of adjacent frames, which considerably reduces the computation burden of frame alignment compared with traditional LSTM-based designs. In addition, the proposed hidden state in BRN is updated by the reused optical flow and refined by the Feature Refinement Module (FRM) for further optimization. Moreover, by utilizing intermediate flow estimation, the proposed method can inference non-linear motion and restore details better. Extensive experiments demonstrate that our optical-flow-reuse-based bidirectional recurrent network (OFR-BRN) is superior to state-of-the-art methods in accuracy and efficiency.