Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robots

TL;DR

This paper introduces PanoMMOcc, a new panoramic multimodal dataset for quadruped robots, and proposes VoxelHound, a perception framework that improves 3D occupancy prediction using multimodal data and novel modules, achieving state-of-the-art results.

Contribution

The paper presents the first real-world panoramic multimodal occupancy dataset for quadruped robots and a novel perception framework with modules designed for spherical imaging and mobility.

Findings

VoxelHound achieves +4.16% mIoU over previous methods.

The dataset enables systematic evaluation of perception in complex environments.

The proposed modules improve volumetric occupancy prediction robustness.

Abstract

Panoramic imagery provides holistic 360{\deg} visual coverage for perception in quadruped robots. However, existing occupancy prediction methods are mainly designed for wheeled autonomous driving and rely heavily on RGB cues, limiting their robustness in complex environments. To bridge this gap, (1) we present PanoMMOcc, the first real-world panoramic multimodal occupancy dataset for quadruped robots, featuring four sensing modalities across diverse scenes. (2) We propose a panoramic multimodal occupancy perception framework, VoxelHound, tailored for legged mobility and spherical imaging. Specifically, we design (i) a Vertical Jitter Compensation (VJC) module to mitigate severe viewpoint perturbations caused by body pitch and roll during mobility, enabling more consistent spatial reasoning, and (ii) an effective Multimodal Information Prompt Fusion (MIPF) module that jointly leverages panoramic visual cues and auxiliary modalities to enhance volumetric occupancy prediction. (3) We establish a benchmark based on PanoMMOcc and provide detailed data analysis to enable systematic evaluation of perception methods under challenging embodied scenarios. Extensive experiments demonstrate that VoxelHound achieves state-of-the-art performance on PanoMMOcc (+4.16%} in mIoU). The dataset and code will be publicly released to facilitate future research on panoramic multimodal 3D perception for embodied robotic systems at https://github.com/SXDR/PanoMMOcc, along with the calibration tools released at https://github.com/losehu/CameraLiDAR-Calib.