LiDAR-Anchored Collaborative Distillation for Robust 2D Representations

TL;DR

This paper introduces a collaborative distillation method that uses LiDAR data as self-supervision to enhance the robustness of 2D image encoders in adverse weather and noisy conditions, while maintaining their original performance.

Contribution

It presents a novel self-supervised learning approach leveraging 3D LiDAR for improving 2D encoder robustness and 3D awareness in challenging environments.

Findings

Outperforms existing methods in various downstream tasks.

Demonstrates strong generalization across diverse conditions.

Enhances 3D awareness from LiDAR data.

Abstract

As deep learning continues to advance, self-supervised learning has made considerable strides. It allows 2D image encoders to extract useful features for various downstream tasks, including those related to vision-based systems. Nevertheless, pre-trained 2D image encoders fall short in conducting the task under noisy and adverse weather conditions beyond clear daytime scenes, which require for robust visual perception. To address these issues, we propose a novel self-supervised approach, \textbf{Collaborative Distillation}, which leverages 3D LiDAR as self-supervision to improve robustness to noisy and adverse weather conditions in 2D image encoders while retaining their original capabilities. Our method outperforms competing methods in various downstream tasks across diverse conditions and exhibits strong generalization ability. In addition, our method also improves 3D awareness stemming from LiDAR's characteristics. This advancement highlights our method's practicality and adaptability in real-world scenarios.