2DDATA: 2D Detection Annotations Transmittable Aggregation for Semantic Segmentation on Point Cloud

TL;DR

This paper introduces 2DDATA, a method to transmit 2D detection annotations to 3D point cloud models, enabling multi-modality fusion without complex calibration or paired data, thus improving semantic segmentation.

Contribution

The paper proposes a novel data-specific branch, Local Object Branch, to transmit 2D bounding box information to 3D models, simplifying multi-modality data fusion.

Findings

Effective transmission of 2D bounding box priors to 3D models.

Fusion of multi-modality data without complex calibration.

Enhanced semantic segmentation performance.

Abstract

Recently, multi-modality models have been introduced because of the complementary information from different sensors such as LiDAR and cameras. It requires paired data along with precise calibrations for all modalities, the complicated calibration among modalities hugely increases the cost of collecting such high-quality datasets, and hinder it from being applied to practical scenarios. Inherit from the previous works, we not only fuse the information from multi-modality without above issues, and also exhaust the information in the RGB modality. We introduced the 2D Detection Annotations Transmittable Aggregation(\textbf{2DDATA}), designing a data-specific branch, called \textbf{Local Object Branch}, which aims to deal with points in a certain bounding box, because of its easiness of acquiring 2D bounding box annotations. We demonstrate that our simple design can transmit bounding box prior information to the 3D encoder model, proving the feasibility of large multi-modality models fused with modality-specific data.