IC-World: In-Context Generation for Shared World Modeling

TL;DR

IC-World introduces a novel framework for shared world modeling that enables parallel video generation from multiple inputs, improving scene and motion consistency through reinforcement learning and novel reward models.

Contribution

It is the first systematic exploration of shared world modeling with video-based models, utilizing in-context generation and reinforcement learning for enhanced consistency.

Findings

Outperforms state-of-the-art in geometry consistency

Outperforms state-of-the-art in motion consistency

Demonstrates effectiveness of reinforcement learning in video generation

Abstract

Video-based world models have recently garnered increasing attention for their ability to synthesize diverse and dynamic visual environments. In this paper, we focus on shared world modeling, where a model generates multiple videos from a set of input images, each representing the same underlying world in different camera poses. We propose IC-World, a novel generation framework, enabling parallel generation for all input images via activating the inherent in-context generation capability of large video models. We further finetune IC-World via reinforcement learning, Group Relative Policy Optimization, together with two proposed novel reward models to enforce scene-level geometry consistency and object-level motion consistency among the set of generated videos. Extensive experiments demonstrate that IC-World substantially outperforms state-of-the-art methods in both geometry and motion consistency. To the best of our knowledge, this is the first work to systematically explore the shared world modeling problem with video-based world models.