See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data

TL;DR

This paper introduces a multi-modal zero-shot point cloud segmentation approach that leverages both point clouds and images to improve recognition of unseen objects, significantly outperforming existing methods.

Contribution

It proposes a novel multi-modal learning framework combining point cloud and image data for zero-shot segmentation, addressing the limited information in point clouds alone.

Findings

Achieved 52% and 49% improvement in unseen class mIoU on SemanticKITTI and nuScenes.

Demonstrated superior performance over state-of-the-art zero-shot segmentation methods.

Validated effectiveness through extensive experiments on two popular benchmarks.

Abstract

Zero-shot point cloud segmentation aims to make deep models capable of recognizing novel objects in point cloud that are unseen in the training phase. Recent trends favor the pipeline which transfers knowledge from seen classes with labels to unseen classes without labels. They typically align visual features with semantic features obtained from word embedding by the supervision of seen classes' annotations. However, point cloud contains limited information to fully match with semantic features. In fact, the rich appearance information of images is a natural complement to the textureless point cloud, which is not well explored in previous literature. Motivated by this, we propose a novel multi-modal zero-shot learning method to better utilize the complementary information of point clouds and images for more accurate visual-semantic alignment. Extensive experiments are performed in two popular benchmarks, i.e., SemanticKITTI and nuScenes, and our method outperforms current SOTA methods with 52% and 49% improvement on average for unseen class mIoU, respectively.