Fast Online Video Super-Resolution with Deformable Attention Pyramid

TL;DR

This paper introduces a fast, efficient online video super-resolution method using a deformable attention pyramid, achieving real-time performance with high accuracy by dynamically focusing on relevant spatial locations.

Contribution

The work presents a novel recurrent VSR architecture with a deformable attention pyramid that reduces computational cost while maintaining high super-resolution quality.

Findings

Outperforms EDVR-M on standard benchmarks

Achieves over 3x speed-up compared to state-of-the-art methods

Reduces processing time and computational complexity

Abstract

Video super-resolution (VSR) has many applications that pose strict causal, real-time, and latency constraints, including video streaming and TV. We address the VSR problem under these settings, which poses additional important challenges since information from future frames is unavailable. Importantly, designing efficient, yet effective frame alignment and fusion modules remain central problems. In this work, we propose a recurrent VSR architecture based on a deformable attention pyramid (DAP). Our DAP aligns and integrates information from the recurrent state into the current frame prediction. To circumvent the computational cost of traditional attention-based methods, we only attend to a limited number of spatial locations, which are dynamically predicted by the DAP. Comprehensive experiments and analysis of the proposed key innovations show the effectiveness of our approach. We significantly reduce processing time and computational complexity in comparison to state-of-the-art methods, while maintaining a high performance. We surpass state-of-the-art method EDVR-M on two standard benchmarks with a speed-up of over $3\times$.