LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training

TL;DR

This paper introduces a LiDAR-camera fusion method that improves panoptic segmentation in autonomous vehicles, achieving higher accuracy without requiring video training data.

Contribution

It proposes a novel feature fusion module and two simple modifications that enhance video panoptic segmentation without additional video training.

Findings

Up to 5-point improvement in segmentation metrics

Effective fusion of LiDAR and image data for scene understanding

High-quality VPS achieved without video training

Abstract

Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied for cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit of 3D data on camera-based scene perception, no specific study has explored the influence of 3D data on image and video panoptic segmentation (VPS).This work seeks to introduce a feature fusion module that enhances PS and VPS by fusing LiDAR and image data for autonomous vehicles. We also illustrate that, in addition to this fusion, our proposed model, which utilizes two simple modifications, can further deliver even more high-quality VPS without being trained on video data. The results demonstrate a substantial improvement in both the image and video panoptic segmentation evaluation metrics by up to 5 points.