Zero-Shot Learning on 3D Point Cloud Objects and Beyond

TL;DR

This paper explores zero-shot learning for 3D point cloud classification, proposing a novel method that improves recognition of unseen classes and extends to various ZSL settings, achieving state-of-the-art results.

Contribution

It introduces a new loss function and approach tailored for 3D ZSL, extending to transductive and GZSL scenarios, and demonstrates its effectiveness across multiple datasets.

Findings

Achieves state-of-the-art ZSL and GZSL performance on 3D datasets.

Proposes a loss function that aligns semantics with point cloud features.

Method is also applicable to 2D image classification.

Abstract

Zero-shot learning, the task of learning to recognize new classes not seen during training, has received considerable attention in the case of 2D image classification. However, despite the increasing ubiquity of 3D sensors, the corresponding 3D point cloud classification problem has not been meaningfully explored and introduces new challenges. In this paper, we identify some of the challenges and apply 2D Zero-Shot Learning (ZSL) methods in the 3D domain to analyze the performance of existing models. Then, we propose a novel approach to address the issues specific to 3D ZSL. We first present an inductive ZSL process and then extend it to the transductive ZSL and Generalized ZSL (GZSL) settings for 3D point cloud classification. To this end, a novel loss function is developed that simultaneously aligns seen semantics with point cloud features and takes advantage of unlabeled test data to address some known issues (e.g., the problems of domain adaptation, hubness, and data bias). While designed for the particularities of 3D point cloud classification, the method is shown to also be applicable to the more common use-case of 2D image classification. An extensive set of experiments is carried out, establishing state-of-the-art for ZSL and GZSL on synthetic (ModelNet40, ModelNet10, McGill) and real (ScanObjectNN) 3D point cloud datasets.