Generalizable Implicit Motion Modeling for Video Frame Interpolation

TL;DR

This paper introduces GIMM, a novel motion modeling approach for video frame interpolation that effectively captures spatiotemporal dynamics by encoding motion priors and predicting flows with neural networks, outperforming current methods.

Contribution

GIMM is a new implicit motion modeling paradigm that encodes motion priors and predicts arbitrary-timestep flows, enhancing VFI performance.

Findings

GIMM outperforms state-of-the-art methods on standard benchmarks.

GIMM effectively models complex spatiotemporal motion dynamics.

The approach integrates seamlessly with existing flow-based VFI models.

Abstract

Motion modeling is critical in flow-based Video Frame Interpolation (VFI). Existing paradigms either consider linear combinations of bidirectional flows or directly predict bilateral flows for given timestamps without exploring favorable motion priors, thus lacking the capability of effectively modeling spatiotemporal dynamics in real-world videos. To address this limitation, in this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel and effective approach to motion modeling for VFI. Specifically, to enable GIMM as an effective motion modeling paradigm, we design a motion encoding pipeline to model spatiotemporal motion latent from bidirectional flows extracted from pre-trained flow estimators, effectively representing input-specific motion priors. Then, we implicitly predict arbitrary-timestep optical flows within two adjacent input frames via an adaptive coordinate-based neural network, with spatiotemporal coordinates and motion latent as inputs. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion. We show that GIMM performs better than the current state of the art on standard VFI benchmarks.