Occupancy World Model for Robots

TL;DR

This paper introduces RoboOccWorld, a novel occupancy world model for indoor robots that predicts scene evolution using a guided autoregressive transformer and spatio-temporal aggregation, outperforming existing methods.

Contribution

The work presents a new framework with CCSA and HSTA for indoor 3D occupancy prediction, and restructures the OccWorld-ScanNet benchmark for better evaluation.

Findings

RoboOccWorld outperforms state-of-the-art methods in indoor 3D occupancy prediction.

The proposed CCSA effectively guides the transformer with camera pose conditions.

HSTA improves the exploitation of spatio-temporal cues from observations.

Abstract

Understanding and forecasting the scene evolutions deeply affect the exploration and decision of embodied agents. While traditional methods simulate scene evolutions through trajectory prediction of potential instances, current works use the occupancy world model as a generative framework for describing fine-grained overall scene dynamics. However, existing methods cluster on the outdoor structured road scenes, while ignoring the exploration of forecasting 3D occupancy scene evolutions for robots in indoor scenes. In this work, we explore a new framework for learning the scene evolutions of observed fine-grained occupancy and propose an occupancy world model based on the combined spatio-temporal receptive field and guided autoregressive transformer to forecast the scene evolutions, called RoboOccWorld. We propose the Conditional Causal State Attention (CCSA), which utilizes camera poses of next state as conditions to guide the autoregressive transformer to adapt and understand the indoor robotics scenarios. In order to effectively exploit the spatio-temporal cues from historical observations, Hybrid Spatio-Temporal Aggregation (HSTA) is proposed to obtain the combined spatio-temporal receptive field based on multi-scale spatio-temporal windows. In addition, we restructure the OccWorld-ScanNet benchmark based on local annotations to facilitate the evaluation of the indoor 3D occupancy scene evolution prediction task. Experimental results demonstrate that our RoboOccWorld outperforms state-of-the-art methods in indoor 3D occupancy scene evolution prediction task. The code will be released soon.