ESP-Zero: Unsupervised enhancement of zero-shot classification for Extremely Sparse Point cloud

TL;DR

This paper introduces ESP-Zero, an unsupervised method to improve zero-shot classification of extremely sparse 3D point clouds by enhancing pre-trained encoders without re-training, using a novel attention layer and self-distillation.

Contribution

The paper proposes a novel fused-cross attention layer and a self-distillation schema to adapt point cloud encoders for sparse data without re-training, improving zero-shot classification performance.

Findings

Outperforms state-of-the-art model adaptation methods.

Effectively enhances zero-shot classification on sparse point clouds.

Maintains alignment between point cloud features and text embeddings.

Abstract

In recent years, zero-shot learning has attracted the focus of many researchers, due to its flexibility and generality. Many approaches have been proposed to achieve the zero-shot classification of the point clouds for 3D object understanding, following the schema of CLIP. However, in the real world, the point clouds could be extremely sparse, dramatically limiting the effectiveness of the 3D point cloud encoders, and resulting in the misalignment of point cloud features and text embeddings. To the point cloud encoders to fit the extremely sparse point clouds without re-running the pre-training procedure which could be time-consuming and expensive, in this work, we propose an unsupervised model adaptation approach to enhance the point cloud encoder for the extremely sparse point clouds. We propose a novel fused-cross attention layer that expands the pre-trained self-attention layer with additional learnable tokens and attention blocks, which effectively modifies the point cloud features while maintaining the alignment between point cloud features and text embeddings. We also propose a complementary learning-based self-distillation schema that encourages the modified features to be pulled apart from the irrelevant text embeddings without overfitting the feature space to the observed text embeddings. Extensive experiments demonstrate that the proposed approach effectively increases the zero-shot capability on extremely sparse point clouds, and overwhelms other state-of-the-art model adaptation approaches.