Learning World Models for Interactive Video Generation

TL;DR

This paper addresses the challenges in long video generation by enhancing world models with retrieval-augmented methods to improve coherence and reduce errors.

Contribution

It introduces VRAG, a retrieval-augmented approach with explicit global state conditioning, to significantly improve long-term spatiotemporal coherence in video generation.

Findings

VRAG reduces long-term compounding errors in video generation.

Explicit global state conditioning enhances spatiotemporal consistency.

Naive autoregressive methods are less effective for long-term video coherence.

Abstract

Foundational world models must be both interactive and preserve spatiotemporal coherence for effective future planning with action choices. However, present models for long video generation have limited inherent world modeling capabilities due to two main challenges: compounding errors and insufficient memory mechanisms. We enhance image-to-video models with interactive capabilities through additional action conditioning and autoregressive framework, and reveal that compounding error is inherently irreducible in autoregressive video generation, while insufficient memory mechanism leads to incoherence of world models. We propose video retrieval augmented generation (VRAG) with explicit global state conditioning, which significantly reduces long-term compounding errors and increases spatiotemporal consistency of world models. In contrast, naive autoregressive generation with extended context windows and retrieval-augmented generation prove less effective for video generation, primarily due to the limited in-context learning capabilities of current video models. Our work illuminates the fundamental challenges in video world models and establishes a comprehensive benchmark for improving video generation models with internal world modeling capabilities.