Back to the Features: DINO as a Foundation for Video World Models

TL;DR

DINO-world is a versatile video world model that predicts future frames in DINOv2's latent space, demonstrating strong performance across diverse scenes and enabling action-conditioned planning.

Contribution

It introduces a pre-trained image encoder-based video world model trained on large-scale uncurated videos, advancing future prediction and planning capabilities.

Findings

Outperforms previous models on video prediction benchmarks

Shows strong understanding of intuitive physics

Enables action-conditioned planning in latent space

Abstract

We present DINO-world, a powerful generalist video world model trained to predict future frames in the latent space of DINOv2. By leveraging a pre-trained image encoder and training a future predictor on a large-scale uncurated video dataset, DINO-world learns the temporal dynamics of diverse scenes, from driving and indoor scenes to simulated environments. We show that DINO-world outperforms previous models on a variety of video prediction benchmarks, e.g. segmentation and depth forecasting, and demonstrates strong understanding of intuitive physics. Furthermore, we show that it is possible to fine-tune the predictor on observation-action trajectories. The resulting action-conditioned world model can be used for planning by simulating candidate trajectories in latent space.