OmniNWM: Omniscient Driving Navigation World Models

TL;DR

OmniNWM is a comprehensive world model for autonomous driving that integrates panoramic video generation, precise control, and rule-based rewards, enabling improved performance and stability in navigation tasks.

Contribution

The paper introduces OmniNWM, a unified framework that jointly models panoramic videos, precise trajectory control, and occupancy-based rewards for autonomous driving.

Findings

Achieves state-of-the-art video generation quality.

Demonstrates high control accuracy and stability.

Provides a reliable evaluation framework with occupancy-grounded rewards.

Abstract

Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. Existing models, however, are typically restricted to limited state modalities, short video sequences, imprecise action control, and a lack of reward awareness. In this paper, we introduce OmniNWM, an omniscient panoramic navigation world model that addresses all three dimensions within a unified framework. For state, OmniNWM jointly generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy. A flexible forcing strategy enables high-quality long-horizon auto-regressive generation. For action, we introduce a normalized panoramic Plucker ray-map representation that encodes input trajectories into pixel-level signals, enabling highly precise and generalizable control over panoramic video generation. Regarding reward, we move beyond learning reward functions with external image-based models: instead, we leverage the generated 3D occupancy to directly define rule-based dense rewards for driving compliance and safety. Extensive experiments demonstrate that OmniNWM achieves state-of-the-art performance in video generation, control accuracy, and long-horizon stability, while providing a reliable closed-loop evaluation framework through occupancy-grounded rewards. Project page is available at https://arlo0o.github.io/OmniNWM/.