PillarNeSt: Embracing Backbone Scaling and Pretraining for Pillar-based 3D Object Detection

TL;DR

This paper demonstrates that scaling and pretraining 2D backbones significantly improve pillar-based 3D object detection, introducing PillarNeSt which leverages pretrained dense ConvNets for better performance.

Contribution

The paper introduces PillarNeSt, a pillar-based 3D detector that uses pretrained 2D ConvNets, showing substantial performance gains over existing methods.

Findings

PillarNeSt outperforms existing 3D detectors on nuScenes and Argoversev2 datasets.

Pretraining 2D backbones enhances feature extraction for 3D detection.

Adaptive design of ConvNets based on point cloud features improves effectiveness.

Abstract

This paper shows the effectiveness of 2D backbone scaling and pretraining for pillar-based 3D object detectors. Pillar-based methods mainly employ randomly initialized 2D convolution neural network (ConvNet) for feature extraction and fail to enjoy the benefits from the backbone scaling and pretraining in the image domain. To show the scaling-up capacity in point clouds, we introduce the dense ConvNet pretrained on large-scale image datasets (e.g., ImageNet) as the 2D backbone of pillar-based detectors. The ConvNets are adaptively designed based on the model size according to the specific features of point clouds, such as sparsity and irregularity. Equipped with the pretrained ConvNets, our proposed pillar-based detector, termed PillarNeSt, outperforms the existing 3D object detectors by a large margin on the nuScenes and Argoversev2 datasets. Our code shall be released upon acceptance.