A Codec Information Assisted Framework for Efficient Compressed Video Super-Resolution

TL;DR

This paper introduces a framework that leverages codec information, such as motion vectors and residuals, to enhance and accelerate video super-resolution models for compressed videos, achieving significant computational savings.

Contribution

The proposed Codec Information Assisted Framework (CIAF) utilizes coded video data to improve efficiency and performance of recurrent VSR models on compressed videos.

Findings

Motion vector based alignment boosts performance with minimal extra computation.

Using residual information allows skipping redundant pixel processing.

Up to 70% computational savings without performance loss on REDS4 videos.

Abstract

Online processing of compressed videos to increase their resolutions attracts increasing and broad attention. Video Super-Resolution (VSR) using recurrent neural network architecture is a promising solution due to its efficient modeling of long-range temporal dependencies. However, state-of-the-art recurrent VSR models still require significant computation to obtain a good performance, mainly because of the complicated motion estimation for frame/feature alignment and the redundant processing of consecutive video frames. In this paper, considering the characteristics of compressed videos, we propose a Codec Information Assisted Framework (CIAF) to boost and accelerate recurrent VSR models for compressed videos. Firstly, the framework reuses the coded video information of Motion Vectors to model the temporal relationships between adjacent frames. Experiments demonstrate that the models with Motion Vector based alignment can significantly boost the performance with negligible additional computation, even comparable to those using more complex optical flow based alignment. Secondly, by further making use of the coded video information of Residuals, the framework can be informed to skip the computation on redundant pixels. Experiments demonstrate that the proposed framework can save up to 70% of the computation without performance drop on the REDS4 test videos encoded by H.264 when CRF is 23.