BF-STVSR: B-Splines and Fourier-Best Friends for High Fidelity Spatial-Temporal Video Super-Resolution

TL;DR

BF-STVSR introduces B-spline and Fourier Mappers to improve continuous spatial-temporal video super-resolution, outperforming prior INR-based methods in detail and temporal consistency.

Contribution

The paper proposes a novel C-STVSR framework with B-spline and Fourier modules, addressing limitations of previous methods in motion representation and detail preservation.

Findings

Achieves state-of-the-art PSNR and SSIM scores

Enhances spatial detail and temporal consistency

Outperforms prior INR-based methods

Abstract

While prior methods in Continuous Spatial-Temporal Video Super-Resolution (C-STVSR) employ Implicit Neural Representation (INR) for continuous encoding, they often struggle to capture the complexity of video data, relying on simple coordinate concatenation and pre-trained optical flow networks for motion representation. Interestingly, we find that adding position encoding, contrary to common observations, does not improve--and even degrades--performance. This issue becomes particularly pronounced when combined with pre-trained optical flow networks, which can limit the model's flexibility. To address these issues, we propose BF-STVSR, a C-STVSR framework with two key modules tailored to better represent spatial and temporal characteristics of video: 1) B-spline Mapper for smooth temporal interpolation, and 2) Fourier Mapper for capturing dominant spatial frequencies. Our approach achieves state-of-the-art in various metrics, including PSNR and SSIM, showing enhanced spatial details and natural temporal consistency. Our code is available https://github.com/Eunjnnn/bfstvsr.