ViPro: Enabling and Controlling Video Prediction for Complex Dynamical Scenarios using Procedural Knowledge

TL;DR

ViPro introduces a novel architecture that integrates procedural domain knowledge into video prediction models, enabling better handling of complex dynamical scenarios and improving downstream control tasks.

Contribution

The paper presents a new architecture that incorporates procedural knowledge directly into video prediction models, addressing limitations of existing methods in complex scenarios.

Findings

State-of-the-art predictors struggle with complex dynamics.

Incorporating procedural knowledge improves prediction accuracy.

The approach facilitates downstream control applications.

Abstract

We propose a novel architecture design for video prediction in order to utilize procedural domain knowledge directly as part of the computational graph of data-driven models. On the basis of new challenging scenarios we show that state-of-the-art video predictors struggle in complex dynamical settings, and highlight that the introduction of prior process knowledge makes their learning problem feasible. Our approach results in the learning of a symbolically addressable interface between data-driven aspects in the model and our dedicated procedural knowledge module, which we utilize in downstream control tasks.