TextOCVP: Object-Centric Video Prediction with Language Guidance

TL;DR

TextOCVP introduces an object-centric video prediction model guided by textual descriptions, enabling more accurate, controllable, and interpretable future scene forecasting in complex environments.

Contribution

It is the first to integrate textual guidance into object-centric video prediction, enhancing scalability, control, and robustness over prior models.

Findings

Outperforms baseline models on two datasets

Provides superior robustness to novel scene configurations

Offers improved controllability and interpretability

Abstract

Understanding and forecasting future scene states is critical for autonomous agents to plan and act effectively in complex environments. Object-centric models, with structured latent spaces, have shown promise in modeling object dynamics and predicting future scene states, but often struggle to scale beyond simple synthetic datasets and to integrate external guidance, limiting their applicability in robotics. To address these limitations, we propose TextOCVP, an object-centric model for video prediction guided by textual descriptions. TextOCVP parses an observed scene into object representations, called slots, and utilizes a text-conditioned transformer predictor to forecast future object states and video frames. Our approach jointly models object dynamics and interactions while incorporating textual guidance, enabling accurate and controllable predictions. TextOCVP's structured latent space offers a more precise control of the forecasting process, outperforming several video prediction baselines on two datasets. Additionally, we show that structured object-centric representations provide superior robustness to novel scene configurations, as well as improved controllability and interpretability, enabling more precise and understandable predictions. Videos and code are available at https://play-slot.github.io/TextOCVP.