Space-Time Video Super-resolution with Neural Operator

TL;DR

This paper introduces a physics-inspired neural operator approach for space-time video super-resolution, effectively handling large motions and surpassing existing methods in accuracy and efficiency.

Contribution

It models MEMC challenges as a continuous function space mapping and employs a Galerkin-type attention mechanism for precise, global motion estimation in ST-VSR.

Findings

Outperforms state-of-the-art ST-VSR methods in accuracy.

Efficiently handles large motions with linear complexity.

Avoids patch partitioning, enabling global receptive fields.

Abstract

This paper addresses the task of space-time video super-resolution (ST-VSR). Existing methods generally suffer from inaccurate motion estimation and motion compensation (MEMC) problems for large motions. Inspired by recent progress in physics-informed neural networks, we model the challenges of MEMC in ST-VSR as a mapping between two continuous function spaces. Specifically, our approach transforms independent low-resolution representations in the coarse-grained continuous function space into refined representations with enriched spatiotemporal details in the fine-grained continuous function space. To achieve efficient and accurate MEMC, we design a Galerkin-type attention function to perform frame alignment and temporal interpolation. Due to the linear complexity of the Galerkin-type attention mechanism, our model avoids patch partitioning and offers global receptive fields, enabling precise estimation of large motions. The experimental results show that the proposed method surpasses state-of-the-art techniques in both fixed-size and continuous space-time video super-resolution tasks.