OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving

TL;DR

OccWorld introduces a novel 3D occupancy-based world model for autonomous driving that captures detailed scene evolution and ego movement, outperforming bounding box methods in efficiency and versatility.

Contribution

This paper presents a new 3D occupancy world model, OccWorld, using a scene tokenizer and transformer to predict scene evolution, offering finer detail and better efficiency than existing box-based methods.

Findings

Effective modeling of scene evolution demonstrated on nuScenes benchmark.

Competitive planning results achieved without instance and map supervision.

Flexible and efficient 3D occupancy representation proven superior to bounding box approaches.

Abstract

Understanding how the 3D scene evolves is vital for making decisions in autonomous driving. Most existing methods achieve this by predicting the movements of object boxes, which cannot capture more fine-grained scene information. In this paper, we explore a new framework of learning a world model, OccWorld, in the 3D Occupancy space to simultaneously predict the movement of the ego car and the evolution of the surrounding scenes. We propose to learn a world model based on 3D occupancy rather than 3D bounding boxes and segmentation maps for three reasons: 1) expressiveness. 3D occupancy can describe the more fine-grained 3D structure of the scene; 2) efficiency. 3D occupancy is more economical to obtain (e.g., from sparse LiDAR points). 3) versatility. 3D occupancy can adapt to both vision and LiDAR. To facilitate the modeling of the world evolution, we learn a reconstruction-based scene tokenizer on the 3D occupancy to obtain discrete scene tokens to describe the surrounding scenes. We then adopt a GPT-like spatial-temporal generative transformer to generate subsequent scene and ego tokens to decode the future occupancy and ego trajectory. Extensive experiments on the widely used nuScenes benchmark demonstrate the ability of OccWorld to effectively model the evolution of the driving scenes. OccWorld also produces competitive planning results without using instance and map supervision. Code: https://github.com/wzzheng/OccWorld.