TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

TL;DR

TaylorSwiftNet is a novel neural network that models video motion using Taylor series, enabling fast, continuous, and adjustable future frame prediction with improved accuracy over traditional RNNs.

Contribution

The paper introduces TaylorSwiftNet, which estimates higher order Taylor series terms for continuous time video prediction, overcoming limitations of discrete models like RNNs.

Findings

Outperforms existing models on multiple datasets.

Predicts future frames in parallel with adjustable temporal resolution.

Demonstrates superior accuracy and flexibility in video forecasting.

Abstract

While recurrent neural networks (RNNs) demonstrate outstanding capabilities for future video frame prediction, they model dynamics in a discrete time space, i.e., they predict the frames sequentially with a fixed temporal step. RNNs are therefore prone to accumulate the error as the number of future frames increases. In contrast, partial differential equations (PDEs) model physical phenomena like dynamics in a continuous time space. However, the estimated PDE for frame forecasting needs to be numerically solved, which is done by discretization of the PDE and diminishes most of the advantages compared to discrete models. In this work, we, therefore, propose to approximate the motion in a video by a continuous function using the Taylor series. To this end, we introduce TaylorSwiftNet, a novel convolutional neural network that learns to estimate the higher order terms of the Taylor series for a given input video. TaylorSwiftNet can swiftly predict future frames in parallel and it allows to change the temporal resolution of the forecast frames on-the-fly. The experimental results on various datasets demonstrate the superiority of our model.