Self-Supervised Image-to-Point Distillation via Semantically Tolerant Contrastive Loss

TL;DR

This paper introduces a semantically tolerant contrastive loss and class balancing technique to improve 3D representation learning from images, effectively addressing self-similarity and class imbalance issues in autonomous driving datasets.

Contribution

It proposes a novel contrastive loss that considers semantic similarity and a class-agnostic balanced loss to enhance 2D-to-3D representation learning for perception tasks.

Findings

Outperforms state-of-the-art methods in 3D semantic segmentation

Improves representation quality across various 2D self-supervised models

Effectively mitigates self-similarity and class imbalance problems

Abstract

An effective framework for learning 3D representations for perception tasks is distilling rich self-supervised image features via contrastive learning. However, image-to point representation learning for autonomous driving datasets faces two main challenges: 1) the abundance of self-similarity, which results in the contrastive losses pushing away semantically similar point and image regions and thus disturbing the local semantic structure of the learned representations, and 2) severe class imbalance as pretraining gets dominated by over-represented classes. We propose to alleviate the self-similarity problem through a novel semantically tolerant image-to-point contrastive loss that takes into consideration the semantic distance between positive and negative image regions to minimize contrasting semantically similar point and image regions. Additionally, we address class imbalance by designing a class-agnostic balanced loss that approximates the degree of class imbalance through an aggregate sample-to-samples semantic similarity measure. We demonstrate that our semantically-tolerant contrastive loss with class balancing improves state-of-the art 2D-to-3D representation learning in all evaluation settings on 3D semantic segmentation. Our method consistently outperforms state-of-the-art 2D-to-3D representation learning frameworks across a wide range of 2D self-supervised pretrained models.