Owl-1: Omni World Model for Consistent Long Video Generation

TL;DR

Owl-1 introduces a novel world modeling approach in a latent space to generate long, coherent videos, overcoming short-term limitations of existing video generation models.

Contribution

It proposes a latent space world model that captures long-term dynamics for consistent long video generation, a significant advancement over frame-by-frame methods.

Findings

Achieves comparable performance with state-of-the-art on VBench datasets.

Enhances long-term consistency and diversity in generated videos.

Validates the effectiveness of latent space modeling for long video synthesis.

Abstract

Video generation models (VGMs) have received extensive attention recently and serve as promising candidates for general-purpose large vision models. While they can only generate short videos each time, existing methods achieve long video generation by iteratively calling the VGMs, using the last-frame output as the condition for the next-round generation. However, the last frame only contains short-term fine-grained information about the scene, resulting in inconsistency in the long horizon. To address this, we propose an Omni World modeL (Owl-1) to produce long-term coherent and comprehensive conditions for consistent long video generation. As videos are observations of the underlying evolving world, we propose to model the long-term developments in a latent space and use VGMs to film them into videos. Specifically, we represent the world with a latent state variable which can be decoded into explicit video observations. These observations serve as a basis for anticipating temporal dynamics which in turn update the state variable. The interaction between evolving dynamics and persistent state enhances the diversity and consistency of the long videos. Extensive experiments show that Owl-1 achieves comparable performance with SOTA methods on VBench-I2V and VBench-Long, validating its ability to generate high-quality video observations. Code: https://github.com/huang-yh/Owl.