RESSCAL3D++: Joint Acquisition and Semantic Segmentation of 3D Point Clouds

TL;DR

This paper introduces RESSCAL3D++, a scalable method for joint acquisition and semantic segmentation of 3D point clouds, leveraging a new dataset VX-S3DIS to demonstrate significant efficiency improvements and early prediction capabilities.

Contribution

The paper presents RESSCAL3D++, an improved scalable approach for 3D point cloud segmentation, and introduces VX-S3DIS, a dataset simulating resolution-scalable 3D sensor behavior.

Findings

Reduces scalability costs from 2% to 0.2% in mIoU.

Achieves speed-ups of 15.6% to 63.9% over non-scalable baseline.

Enables early predictions after only 7% of total inference time.

Abstract

3D scene understanding is crucial for facilitating seamless interaction between digital devices and the physical world. Real-time capturing and processing of the 3D scene are essential for achieving this seamless integration. While existing approaches typically separate acquisition and processing for each frame, the advent of resolution-scalable 3D sensors offers an opportunity to overcome this paradigm and fully leverage the otherwise wasted acquisition time to initiate processing. In this study, we introduce VX-S3DIS, a novel point cloud dataset accurately simulating the behavior of a resolution-scalable 3D sensor. Additionally, we present RESSCAL3D++, an important improvement over our prior work, RESSCAL3D, by incorporating an update module and processing strategy. By applying our method to the new dataset, we practically demonstrate the potential of joint acquisition and semantic segmentation of 3D point clouds. Our resolution-scalable approach significantly reduces scalability costs from 2% to just 0.2% in mIoU while achieving impressive speed-ups of 15.6 to 63.9% compared to the non-scalable baseline. Furthermore, our scalable approach enables early predictions, with the first one occurring after only 7% of the total inference time of the baseline. The new VX-S3DIS dataset is available at https://github.com/remcoroyen/vx-s3dis.