Cascaded Temporal Updating Network for Efficient Video Super-Resolution

TL;DR

This paper introduces a cascaded temporal updating network (CTUN) for efficient video super-resolution that balances high-quality reconstruction with reduced model size and inference time, suitable for resource-limited devices.

Contribution

The paper proposes a novel cascaded alignment and unidirectional propagation framework that significantly reduces parameters and inference time while maintaining or improving super-resolution performance.

Findings

Achieves better results than BasicVSR with only 30% of parameters and runtime.

Develops an implicit cascaded alignment module for efficient spatio-temporal correspondence.

Introduces a hidden updater leveraging future information to enhance efficiency.

Abstract

Existing video super-resolution (VSR) methods generally adopt a recurrent propagation network to extract spatio-temporal information from the entire video sequences, exhibiting impressive performance. However, the key components in recurrent-based VSR networks significantly impact model efficiency, e.g., the alignment module occupies a substantial portion of model parameters, while the bidirectional propagation mechanism significantly amplifies the inference time. Consequently, developing a compact and efficient VSR method that can be deployed on resource-constrained devices, e.g., smartphones, remains challenging. To this end, we propose a cascaded temporal updating network (CTUN) for efficient VSR. We first develop an implicit cascaded alignment module to explore spatio-temporal correspondences from adjacent frames. Moreover, we propose a unidirectional propagation updating network to efficiently explore long-range temporal information, which is crucial for high-quality video reconstruction. Specifically, we develop a simple yet effective hidden updater that can leverage future information to update hidden features during forward propagation, significantly reducing inference time while maintaining performance. Finally, we formulate all of these components into an end-to-end trainable VSR network. Extensive experimental results show that our CTUN achieves a favorable trade-off between efficiency and performance compared to existing methods. Notably, compared with BasicVSR, our method obtains better results while employing only about 30% of the parameters and running time. The source code and pre-trained models will be available at https://github.com/House-Leo/CTUN.