Compositional Video Synthesis by Temporal Object-Centric Learning

TL;DR

This paper introduces a new object-centric framework for video synthesis that achieves high-quality, temporally coherent videos with editable object components, advancing the state-of-the-art in controllable video generation.

Contribution

It extends object-centric learning from images to videos, incorporating temporal dynamics and diffusion models for improved synthesis and editing capabilities.

Findings

Sets new benchmarks in video quality and temporal coherence.

Enables intuitive object editing like insertion, deletion, and replacement.

Maintains consistent object identities across frames.

Abstract

We present a novel framework for compositional video synthesis that leverages temporally consistent object-centric representations, extending our previous work, SlotAdapt, from images to video. While existing object-centric approaches either lack generative capabilities entirely or treat video sequences holistically, thus neglecting explicit object-level structure, our approach explicitly captures temporal dynamics by learning pose invariant object-centric slots and conditioning them on pretrained diffusion models. This design enables high-quality, pixel-level video synthesis with superior temporal coherence, and offers intuitive compositional editing capabilities such as object insertion, deletion, or replacement, maintaining consistent object identities across frames. Extensive experiments demonstrate that our method sets new benchmarks in video generation quality and temporal consistency, outperforming previous object-centric generative methods. Although our segmentation performance closely matches state-of-the-art methods, our approach uniquely integrates this capability with robust generative performance, significantly advancing interactive and controllable video generation and opening new possibilities for advanced content creation, semantic editing, and dynamic scene understanding.