Dreamweaver: Learning Compositional World Models from Pixels

TL;DR

Dreamweaver introduces a neural architecture that learns hierarchical, compositional representations from raw videos, enabling the generation of novel, recombined future scenes without auxiliary data, advancing video world modeling and imagination.

Contribution

It proposes a novel RBSU module and a multi-future-frame prediction objective for unsupervised learning of compositional video representations.

Findings

Outperforms state-of-the-art in world modeling benchmarks.

Enables recombination of learned attributes for novel video generation.

Demonstrates effective disentanglement of static and dynamic concepts.

Abstract

Humans have an innate ability to decompose their perceptions of the world into objects and their attributes, such as colors, shapes, and movement patterns. This cognitive process enables us to imagine novel futures by recombining familiar concepts. However, replicating this ability in artificial intelligence systems has proven challenging, particularly when it comes to modeling videos into compositional concepts and generating unseen, recomposed futures without relying on auxiliary data, such as text, masks, or bounding boxes. In this paper, we propose Dreamweaver, a neural architecture designed to discover hierarchical and compositional representations from raw videos and generate compositional future simulations. Our approach leverages a novel Recurrent Block-Slot Unit (RBSU) to decompose videos into their constituent objects and attributes. In addition, Dreamweaver uses a multi-future-frame prediction objective to capture disentangled representations for dynamic concepts more effectively as well as static concepts. In experiments, we demonstrate our model outperforms current state-of-the-art baselines for world modeling when evaluated under the DCI framework across multiple datasets. Furthermore, we show how the modularized concept representations of our model enable compositional imagination, allowing the generation of novel videos by recombining attributes from previously seen objects. cun-bjy.github.io/dreamweaver-website