Compressed-Domain-Aware Online Video Super-Resolution

TL;DR

This paper introduces CDA-VSR, a novel online video super-resolution method that leverages compressed-domain information to improve efficiency and accuracy, achieving real-time performance and surpassing state-of-the-art results.

Contribution

The paper proposes a compressed-domain-aware network with motion-vector-guided alignment, residual map fusion, and frame-type-aware reconstruction for efficient online VSR.

Findings

Outperforms state-of-the-art methods on REDS4 dataset

Achieves over double the inference speed of previous methods

Improves PSNR by up to 0.13 dB

Abstract

In bandwidth-limited online video streaming, videos are usually downsampled and compressed. Although recent online video super-resolution (online VSR) approaches achieve promising results, they are still compute-intensive and fall short of real-time processing at higher resolutions, due to complex motion estimation for alignment and redundant processing of consecutive frames. To address these issues, we propose a compressed-domain-aware network (CDA-VSR) for online VSR, which utilizes compressed-domain information, including motion vectors, residual maps, and frame types to balance quality and efficiency. Specifically, we propose a motion-vector-guided deformable alignment module that uses motion vectors for coarse warping and learns only local residual offsets for fine-tuned adjustments, thereby maintaining accuracy while reducing computation. Then, we utilize a residual map gated fusion module to derive spatial weights from residual maps, suppressing mismatched regions and emphasizing reliable details. Further, we design a frame-type-aware reconstruction module for adaptive compute allocation across frame types, balancing accuracy and efficiency. On the REDS4 dataset, our CDA-VSR surpasses the state-of-the-art method TMP, with a maximum PSNR improvement of 0.13 dB while delivering more than double the inference speed. The code will be released at https://github.com/sspBIT/CDA-VSR.