UniPAD: A Universal Pre-training Paradigm for Autonomous Driving

TL;DR

UniPAD introduces a versatile self-supervised pre-training paradigm for autonomous driving that leverages 3D volumetric differentiable rendering to improve scene understanding across various 3D tasks.

Contribution

It presents a novel 3D pre-training method using differentiable rendering, enhancing feature learning for autonomous driving beyond traditional 2D-based approaches.

Findings

Significantly improves baseline performance on lidar, camera, and combined data.

Achieves state-of-the-art results on nuScenes 3D object detection and segmentation.

Demonstrates effective integration into both 2D and 3D frameworks.

Abstract

In the context of autonomous driving, the significance of effective feature learning is widely acknowledged. While conventional 3D self-supervised pre-training methods have shown widespread success, most methods follow the ideas originally designed for 2D images. In this paper, we present UniPAD, a novel self-supervised learning paradigm applying 3D volumetric differentiable rendering. UniPAD implicitly encodes 3D space, facilitating the reconstruction of continuous 3D shape structures and the intricate appearance characteristics of their 2D projections. The flexibility of our method enables seamless integration into both 2D and 3D frameworks, enabling a more holistic comprehension of the scenes. We manifest the feasibility and effectiveness of UniPAD by conducting extensive experiments on various downstream 3D tasks. Our method significantly improves lidar-, camera-, and lidar-camera-based baseline by 9.1, 7.7, and 6.9 NDS, respectively. Notably, our pre-training pipeline achieves 73.2 NDS for 3D object detection and 79.4 mIoU for 3D semantic segmentation on the nuScenes validation set, achieving state-of-the-art results in comparison with previous methods. The code will be available at https://github.com/Nightmare-n/UniPAD.