Video Super-Resolution with Long-Term Self-Exemplars

TL;DR

This paper introduces a video super-resolution approach that exploits long-term, cross-scale self-exemplars across distant frames, significantly improving super-resolution quality by leveraging redundant information often overlooked by existing methods.

Contribution

The paper proposes a novel long-term cross-scale aggregation method with a multi-reference alignment module for enhanced video super-resolution, along with a practical training strategy.

Findings

Outperforms state-of-the-art methods on CarCam and Waymo datasets

Effectively leverages distant frame information for super-resolution

Demonstrates significant quality improvements in experimental results

Abstract

Existing video super-resolution methods often utilize a few neighboring frames to generate a higher-resolution image for each frame. However, the redundant information between distant frames has not been fully exploited in these methods: corresponding patches of the same instance appear across distant frames at different scales. Based on this observation, we propose a video super-resolution method with long-term cross-scale aggregation that leverages similar patches (self-exemplars) across distant frames. Our model also consists of a multi-reference alignment module to fuse the features derived from similar patches: we fuse the features of distant references to perform high-quality super-resolution. We also propose a novel and practical training strategy for referenced-based super-resolution. To evaluate the performance of our proposed method, we conduct extensive experiments on our collected CarCam dataset and the Waymo Open dataset, and the results demonstrate our method outperforms state-of-the-art methods. Our source code will be publicly available.