3DGeoDet: General-purpose Geometry-aware Image-based 3D Object Detection

TL;DR

3DGeoDet introduces a geometry-aware 3D object detection method that effectively uses predicted depth to generate explicit and implicit 3D representations, significantly improving accuracy in diverse environments without requiring 3D supervision.

Contribution

The paper presents a novel approach combining explicit voxel occupancy and implicit TSDF representations for improved 3D detection from images, without needing 3D supervision.

Findings

Achieved 9.3 mAP@0.5 on SUN RGB-D

Improved 3.3 mAP@0.5 on ScanNetV2

Enhanced AP3D@0.7 on KITTI

Abstract

This paper proposes 3DGeoDet, a novel geometry-aware 3D object detection approach that effectively handles single- and multi-view RGB images in indoor and outdoor environments, showcasing its general-purpose applicability. The key challenge for image-based 3D object detection tasks is the lack of 3D geometric cues, which leads to ambiguity in establishing correspondences between images and 3D representations. To tackle this problem, 3DGeoDet generates efficient 3D geometric representations in both explicit and implicit manners based on predicted depth information. Specifically, we utilize the predicted depth to learn voxel occupancy and optimize the voxelized 3D feature volume explicitly through the proposed voxel occupancy attention. To further enhance 3D awareness, the feature volume is integrated with an implicit 3D representation, the truncated signed distance function (TSDF). Without requiring supervision from 3D signals, we significantly improve the model's comprehension of 3D geometry by leveraging intermediate 3D representations and achieve end-to-end training. Our approach surpasses the performance of state-of-the-art image-based methods on both single- and multi-view benchmark datasets across diverse environments, achieving a 9.3 mAP@0.5 improvement on the SUN RGB-D dataset, a 3.3 mAP@0.5 improvement on the ScanNetV2 dataset, and a 0.19 AP3D@0.7 improvement on the KITTI dataset. The project page is available at: https://cindy0725.github.io/3DGeoDet/.