Learning Depth-Guided Convolutions for Monocular 3D Object Detection

TL;DR

This paper introduces D4LCN, a novel depth-guided local convolutional network that enhances monocular 3D object detection by learning adaptive filters from depth maps, outperforming existing methods significantly.

Contribution

It proposes a new depth-guided local convolutional network (D4LCN) that automatically learns filters from depth maps, bridging the gap between 2D image features and 3D structure for improved detection.

Findings

D4LCN outperforms existing methods by large margins.

Achieves 9.1% improvement over state-of-the-art on KITTI.

Extensive experiments validate the effectiveness of D4LCN.

Abstract

3D object detection from a single image without LiDAR is a challenging task due to the lack of accurate depth information. Conventional 2D convolutions are unsuitable for this task because they fail to capture local object and its scale information, which are vital for 3D object detection. To better represent 3D structure, prior arts typically transform depth maps estimated from 2D images into a pseudo-LiDAR representation, and then apply existing 3D point-cloud based object detectors. However, their results depend heavily on the accuracy of the estimated depth maps, resulting in suboptimal performance. In this work, instead of using pseudo-LiDAR representation, we improve the fundamental 2D fully convolutions by proposing a new local convolutional network (LCN), termed Depth-guided Dynamic-Depthwise-Dilated LCN (D$^4$LCN), where the filters and their receptive fields can be automatically learned from image-based depth maps, making different pixels of different images have different filters. D$^4$LCN overcomes the limitation of conventional 2D convolutions and narrows the gap between image representation and 3D point cloud representation. Extensive experiments show that D$^4$LCN outperforms existing works by large margins. For example, the relative improvement of D$^4$LCN against the state-of-the-art on KITTI is 9.1\% in the moderate setting. The code is available at https://github.com/dingmyu/D4LCN.