OCK: Unsupervised Dynamic Video Prediction with Object-Centric Kinematics

TL;DR

The paper introduces OCK, an unsupervised model that enhances dynamic video prediction by explicitly modeling object kinematics alongside appearance, improving understanding of complex multi-object scenes.

Contribution

The paper presents a novel object kinematics component integrated into object-centric transformers, enabling better modeling of object motion dynamics in video prediction.

Findings

Superior performance on complex scene prediction tasks

Effective modeling of object motion and interactions

Long-term spatiotemporal prediction accuracy

Abstract

Human perception involves decomposing complex multi-object scenes into time-static object appearance (i.e., size, shape, color) and time-varying object motion (i.e., position, velocity, acceleration). For machines to achieve human-like intelligence in real-world interactions, understanding these physical properties of objects is essential, forming the foundation for dynamic video prediction. While recent advancements in object-centric transformers have demonstrated potential in video prediction, they primarily focus on object appearance, often overlooking motion dynamics, which is crucial for modeling dynamic interactions and maintaining temporal consistency in complex environments. To address these limitations, we propose OCK, a dynamic video prediction model leveraging object-centric kinematics and object slots. We introduce a novel component named Object Kinematics that comprises explicit object motions, serving as an additional attribute beyond conventional appearance features to model dynamic scenes. The Object Kinematics are integrated into various OCK mechanisms, enabling spatiotemporal prediction of complex object interactions over long video sequences. Our model demonstrates superior performance in handling complex scenes with intricate object attributes and motions, highlighting its potential for applicability in vision-related dynamics learning tasks.