Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos

TL;DR

This paper introduces a point-level contrastive learning framework with semantic clustering for point cloud videos, capturing fine-grained semantics and improving downstream task performance.

Contribution

It presents a unified self-supervised learning method at the point level with semantic alignment of superpoints, addressing limitations of clip/frame-level approaches.

Findings

Outperforms supervised methods on various downstream tasks

Effectively captures multi-scale semantic cues

Demonstrates superior transferability of learned representations

Abstract

We propose a unified point cloud video self-supervised learning framework for object-centric and scene-centric data. Previous methods commonly conduct representation learning at the clip or frame level and cannot well capture fine-grained semantics. Instead of contrasting the representations of clips or frames, in this paper, we propose a unified self-supervised framework by conducting contrastive learning at the point level. Moreover, we introduce a new pretext task by achieving semantic alignment of superpoints, which further facilitates the representations to capture semantic cues at multiple scales. In addition, due to the high redundancy in the temporal dimension of dynamic point clouds, directly conducting contrastive learning at the point level usually leads to massive undesired negatives and insufficient modeling of positive representations. To remedy this, we propose a selection strategy to retain proper negatives and make use of high-similarity samples from other instances as positive supplements. Extensive experiments show that our method outperforms supervised counterparts on a wide range of downstream tasks and demonstrates the superior transferability of the learned representations.