VideoWorld: Exploring Knowledge Learning from Unlabeled Videos

TL;DR

VideoWorld demonstrates that a deep generative model trained solely on unlabeled videos can acquire complex knowledge and skills, rivaling methods that rely on labeled data or reinforcement learning, by focusing on visual change representation.

Contribution

The paper introduces VideoWorld, a novel auto-regressive video generation model that learns knowledge from unlabeled videos, highlighting the importance of visual change representation and achieving strong performance without reinforcement learning.

Findings

VideoWorld reaches 5-dan level in Video-GoBench with 300M parameters.

The model effectively learns control operations and generalizes across environments.

Visual change representation is crucial for knowledge acquisition.

Abstract

This work explores whether a deep generative model can learn complex knowledge solely from visual input, in contrast to the prevalent focus on text-based models like large language models (LLMs). We develop VideoWorld, an auto-regressive video generation model trained on unlabeled video data, and test its knowledge acquisition abilities in video-based Go and robotic control tasks. Our experiments reveal two key findings: (1) video-only training provides sufficient information for learning knowledge, including rules, reasoning and planning capabilities, and (2) the representation of visual change is crucial for knowledge acquisition. To improve both the efficiency and efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional level in the Video-GoBench with just a 300-million-parameter model, without relying on search algorithms or reward mechanisms typical in reinforcement learning. In robotic tasks, VideoWorld effectively learns diverse control operations and generalizes across environments, approaching the performance of oracle models in CALVIN and RLBench. This study opens new avenues for knowledge acquisition from visual data, with all code, data, and models open-sourced for further research.