LMVP: Video Predictor with Leaked Motion Information

TL;DR

LMVP introduces a novel video prediction model that leverages leaked motion information to effectively learn and generate future frames with high spatio-temporal consistency, achieving state-of-the-art results.

Contribution

The paper proposes a Leaked Motion Video Predictor (LMVP) with a unique motion guider that learns and guides motion prediction without human labels.

Findings

Achieves state-of-the-art performance on synthetic and real datasets.

Effectively models static and temporal features in videos.

Leverages leaked information to improve motion learning.

Abstract

We propose a Leaked Motion Video Predictor (LMVP) to predict future frames by capturing the spatial and temporal dependencies from given inputs. The motion is modeled by a newly proposed component, motion guider, which plays the role of both learner and teacher. Specifically, it {\em learns} the temporal features from real data and {\em guides} the generator to predict future frames. The spatial consistency in video is modeled by an adaptive filtering network. To further ensure the spatio-temporal consistency of the prediction, a discriminator is also adopted to distinguish the real and generated frames. Further, the discriminator leaks information to the motion guider and the generator to help the learning of motion. The proposed LMVP can effectively learn the static and temporal features in videos without the need for human labeling. Experiments on synthetic and real data demonstrate that LMVP can yield state-of-the-art results.