4D-Net for Learned Multi-Modal Alignment

TL;DR

4D-Net is a novel 3D object detection method that leverages 4D data from point clouds and RGB over time, using dynamic connection learning and geometric constraints to improve detection accuracy, especially for distant objects.

Contribution

The paper introduces 4D-Net, a new approach that effectively integrates temporal 3D and RGB data with dynamic feature connections and geometric constraints for enhanced detection.

Findings

Outperforms state-of-the-art on Waymo dataset

Better utilizes motion cues and dense image info

Improves detection of distant objects

Abstract

We present 4D-Net, a 3D object detection approach, which utilizes 3D Point Cloud and RGB sensing information, both in time. We are able to incorporate the 4D information by performing a novel dynamic connection learning across various feature representations and levels of abstraction, as well as by observing geometric constraints. Our approach outperforms the state-of-the-art and strong baselines on the Waymo Open Dataset. 4D-Net is better able to use motion cues and dense image information to detect distant objects more successfully.