MoTIF: Learning Motion Trajectories with Local Implicit Neural Functions for Continuous Space-Time Video Super-Resolution

TL;DR

MoTIF introduces a novel space-time local implicit neural function to learn forward motion trajectories for continuous space-time video super-resolution, achieving state-of-the-art results by effectively propagating temporal information.

Contribution

The paper proposes MoTIF, a new framework that models forward motion trajectories with local implicit neural functions for improved continuous space-time video super-resolution.

Findings

Achieves state-of-the-art performance on C-STVSR benchmarks.

Effectively propagates temporal information using learned forward motion.

Outperforms existing methods in quality and flexibility.

Abstract

This work addresses continuous space-time video super-resolution (C-STVSR) that aims to up-scale an input video both spatially and temporally by any scaling factors. One key challenge of C-STVSR is to propagate information temporally among the input video frames. To this end, we introduce a space-time local implicit neural function. It has the striking feature of learning forward motion for a continuum of pixels. We motivate the use of forward motion from the perspective of learning individual motion trajectories, as opposed to learning a mixture of motion trajectories with backward motion. To ease motion interpolation, we encode sparsely sampled forward motion extracted from the input video as the contextual input. Along with a reliability-aware splatting and decoding scheme, our framework, termed MoTIF, achieves the state-of-the-art performance on C-STVSR. The source code of MoTIF is available at https://github.com/sichun233746/MoTIF.