Enhancing Video Super-Resolution via Implicit Resampling-based Alignment

TL;DR

This paper introduces an implicit resampling-based alignment method for video super-resolution that preserves high-frequency details better than traditional bilinear interpolation, leading to improved performance.

Contribution

It proposes a novel implicit resampling approach using sinusoidal encoding and coordinate networks to enhance alignment in video super-resolution.

Findings

Improves super-resolution quality on synthetic and real datasets.

Outperforms traditional bilinear resampling methods.

Enhances state-of-the-art frameworks with minimal computational overhead.

Abstract

In video super-resolution, it is common to use a frame-wise alignment to support the propagation of information over time. The role of alignment is well-studied for low-level enhancement in video, but existing works overlook a critical step -- resampling. We show through extensive experiments that for alignment to be effective, the resampling should preserve the reference frequency spectrum while minimizing spatial distortions. However, most existing works simply use a default choice of bilinear interpolation for resampling even though bilinear interpolation has a smoothing effect and hinders super-resolution. From these observations, we propose an implicit resampling-based alignment. The sampling positions are encoded by a sinusoidal positional encoding, while the value is estimated with a coordinate network and a window-based cross-attention. We show that bilinear interpolation inherently attenuates high-frequency information while an MLP-based coordinate network can approximate more frequencies. Experiments on synthetic and real-world datasets show that alignment with our proposed implicit resampling enhances the performance of state-of-the-art frameworks with minimal impact on both compute and parameters.