Real-World Video for Zoom Enhancement based on Spatio-Temporal Coupling

TL;DR

This paper introduces a new real-world video dataset and a spatio-temporal coupling loss to improve zoom quality in video super-resolution, demonstrating superior results in various zoom scenarios.

Contribution

The paper presents a real-world video benchmark and a novel spatio-temporal coupling loss for enhanced zoom quality in video super-resolution.

Findings

The proposed method outperforms existing models in zoom scenarios.

The real-world dataset VideoRAW enables more realistic training.

Spatio-temporal coupling improves feature alignment and fusion.

Abstract

In recent years, single-frame image super-resolution (SR) has become more realistic by considering the zooming effect and using real-world short- and long-focus image pairs. In this paper, we further investigate the feasibility of applying realistic multi-frame clips to enhance zoom quality via spatio-temporal information coupling. Specifically, we first built a real-world video benchmark, VideoRAW, by a synchronized co-axis optical system. The dataset contains paired short-focus raw and long-focus sRGB videos of different dynamic scenes. Based on VideoRAW, we then presented a Spatio-Temporal Coupling Loss, termed as STCL. The proposed STCL is intended for better utilization of information from paired and adjacent frames to align and fuse features both temporally and spatially at the feature level. The outperformed experimental results obtained in different zoom scenarios demonstrate the superiority of integrating real-world video dataset and STCL into existing SR models for zoom quality enhancement, and reveal that the proposed method can serve as an advanced and viable tool for video zoom.