Online panoptic 3D reconstruction as a Linear Assignment Problem

TL;DR

This paper presents a real-time, online method for 3D scene reconstruction from panoptic image segmentations, utilizing a linear assignment approach to improve scalability and processing speed for large environments.

Contribution

The authors introduce a simple data-association algorithm that enables real-time, online 3D reconstruction from panoptic segmentation, outperforming previous methods in speed and scalability.

Findings

Outperforms earlier online 3D reconstruction methods

Achieves high frame rates suitable for real-time applications

Scalable to large environments

Abstract

Real-time holistic scene understanding would allow machines to interpret their surrounding in a much more detailed manner than is currently possible. While panoptic image segmentation methods have brought image segmentation closer to this goal, this information has to be described relative to the 3D environment for the machine to be able to utilise it effectively. In this paper, we investigate methods for sequentially reconstructing static environments from panoptic image segmentations in 3D. We specifically target real-time operation: the algorithm must process data strictly online and be able to run at relatively fast frame rates. Additionally, the method should be scalable for environments large enough for practical applications. By applying a simple but powerful data-association algorithm, we outperform earlier similar works when operating purely online. Our method is also capable of reaching frame-rates high enough for real-time applications and is scalable to larger environments as well. Source code and further demonstrations are released to the public at: \url{https://tutvision.github.io/Online-Panoptic-3D/}