VideoPCDNet: Video Parsing and Prediction with Phase Correlation Networks

TL;DR

VideoPCDNet introduces an unsupervised, object-centric framework for video parsing and prediction using frequency-domain phase correlation, enabling accurate tracking and future frame prediction with interpretable representations.

Contribution

It presents a novel frequency-domain approach for unsupervised object decomposition and motion modeling in videos, improving tracking and prediction accuracy.

Findings

Outperforms baseline models on synthetic datasets.

Learns interpretable object and motion representations.

Enables accurate unsupervised object tracking and prediction.

Abstract

Understanding and predicting video content is essential for planning and reasoning in dynamic environments. Despite advancements, unsupervised learning of object representations and dynamics remains challenging. We present VideoPCDNet, an unsupervised framework for object-centric video decomposition and prediction. Our model uses frequency-domain phase correlation techniques to recursively parse videos into object components, which are represented as transformed versions of learned object prototypes, enabling accurate and interpretable tracking. By explicitly modeling object motion through a combination of frequency domain operations and lightweight learned modules, VideoPCDNet enables accurate unsupervised object tracking and prediction of future video frames. In our experiments, we demonstrate that VideoPCDNet outperforms multiple object-centric baseline models for unsupervised tracking and prediction on several synthetic datasets, while learning interpretable object and motion representations.