A Closer Look at Invariances in Self-supervised Pre-training for 3D Vision

TL;DR

This paper systematically compares different invariances used in self-supervised 3D vision pre-training within a unified framework, revealing their contributions and proposing a contrastive learning method that improves downstream task performance.

Contribution

It introduces the first unified framework for fair comparison of invariances in 3D pre-training and proposes a contrastive learning method that enhances model performance.

Findings

Different invariances contribute variably to pre-training effectiveness.

The proposed contrastive learning method significantly improves downstream task results.

Pre-trained VoteNet outperforms previous methods on SUN RGB-D and ScanNet benchmarks.

Abstract

Self-supervised pre-training for 3D vision has drawn increasing research interest in recent years. In order to learn informative representations, a lot of previous works exploit invariances of 3D features, e.g., perspective-invariance between views of the same scene, modality-invariance between depth and RGB images, format-invariance between point clouds and voxels. Although they have achieved promising results, previous researches lack a systematic and fair comparison of these invariances. To address this issue, our work, for the first time, introduces a unified framework, under which various pre-training methods can be investigated. We conduct extensive experiments and provide a closer look at the contributions of different invariances in 3D pre-training. Also, we propose a simple but effective method that jointly pre-trains a 3D encoder and a depth map encoder using contrastive learning. Models pre-trained with our method gain significant performance boost in downstream tasks. For instance, a pre-trained VoteNet outperforms previous methods on SUN RGB-D and ScanNet object detection benchmarks with a clear margin.