Learning Robust 3D Representation from CLIP via Dual Denoising

TL;DR

This paper introduces Dual Denoising, a framework that enhances the robustness and generalization of 3D representations learned from CLIP, especially against adversarial attacks, without requiring adversarial training.

Contribution

It proposes a novel dual denoising framework combining a proxy task and feature denoising network for robust 3D pre-training from CLIP.

Findings

Improves 3D representation performance in zero-shot settings

Enhances adversarial robustness without adversarial training

Effective in cross-domain point cloud generalization

Abstract

In this paper, we explore a critical yet under-investigated issue: how to learn robust and well-generalized 3D representation from pre-trained vision language models such as CLIP. Previous works have demonstrated that cross-modal distillation can provide rich and useful knowledge for 3D data. However, like most deep learning models, the resultant 3D learning network is still vulnerable to adversarial attacks especially the iterative attack. In this work, we propose Dual Denoising, a novel framework for learning robust and well-generalized 3D representations from CLIP. It combines a denoising-based proxy task with a novel feature denoising network for 3D pre-training. Additionally, we propose utilizing parallel noise inference to enhance the generalization of point cloud features under cross domain settings. Experiments show that our model can effectively improve the representation learning performance and adversarial robustness of the 3D learning network under zero-shot settings without adversarial training. Our code is available at https://github.com/luoshuqing2001/Dual_Denoising.