COME: Adding Scene-Centric Forecasting Control to Occupancy World Model

TL;DR

COME introduces a scene-centric forecasting control framework that improves occupancy prediction accuracy by disentangling ego-motion from environmental changes, leading to significant performance gains on the nuScenes-Occ3D dataset.

Contribution

The paper presents a novel scene-centric approach that integrates forecasting control into the occupancy world model, enhancing prediction accuracy and controllability.

Findings

Achieves 26.3% better mIoU than DOME.

Achieves 23.7% better mIoU than UniScene.

Demonstrates consistent improvements across various configurations.

Abstract

World models are critical for autonomous driving to simulate environmental dynamics and generate synthetic data. Existing methods struggle to disentangle ego-vehicle motion (perspective shifts) from scene evolvement (agent interactions), leading to suboptimal predictions. Instead, we propose to separate environmental changes from ego-motion by leveraging the scene-centric coordinate systems. In this paper, we introduce COME: a framework that integrates scene-centric forecasting Control into the Occupancy world ModEl. Specifically, COME first generates ego-irrelevant, spatially consistent future features through a scene-centric prediction branch, which are then converted into scene condition using a tailored ControlNet. These condition features are subsequently injected into the occupancy world model, enabling more accurate and controllable future occupancy predictions. Experimental results on the nuScenes-Occ3D dataset show that COME achieves consistent and significant improvements over state-of-the-art (SOTA) methods across diverse configurations, including different input sources (ground-truth, camera-based, fusion-based occupancy) and prediction horizons (3s and 8s). For example, under the same settings, COME achieves 26.3% better mIoU metric than DOME and 23.7% better mIoU metric than UniScene. These results highlight the efficacy of disentangled representation learning in enhancing spatio-temporal prediction fidelity for world models. Code and videos will be available at https://github.com/synsin0/COME.