Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation

TL;DR

This paper introduces a geometry-aware fusion network that combines stereo camera images and LiDAR point clouds in a unified 3D volume for accurate long-range depth estimation, achieving state-of-the-art results.

Contribution

The proposed volumetric propagation network uniquely embeds point clouds into a 3D volume and introduces FusionConv for enhanced feature extraction, improving stereo-LiDAR fusion performance.

Findings

Achieves state-of-the-art results on KITTI and Virtual-KITTI datasets.

Effectively reduces uncertainty in stereo-LiDAR correspondence.

Enhances long-range depth estimation accuracy.

Abstract

Stereo-LiDAR fusion is a promising task in that we can utilize two different types of 3D perceptions for practical usage -- dense 3D information (stereo cameras) and highly-accurate sparse point clouds (LiDAR). However, due to their different modalities and structures, the method of aligning sensor data is the key for successful sensor fusion. To this end, we propose a geometry-aware stereo-LiDAR fusion network for long-range depth estimation, called volumetric propagation network. The key idea of our network is to exploit sparse and accurate point clouds as a cue for guiding correspondences of stereo images in a unified 3D volume space. Unlike existing fusion strategies, we directly embed point clouds into the volume, which enables us to propagate valid information into nearby voxels in the volume, and to reduce the uncertainty of correspondences. Thus, it allows us to fuse two different input modalities seamlessly and regress a long-range depth map. Our fusion is further enhanced by a newly proposed feature extraction layer for point clouds guided by images: FusionConv. FusionConv extracts point cloud features that consider both semantic (2D image domain) and geometric (3D domain) relations and aid fusion at the volume. Our network achieves state-of-the-art performance on the KITTI and the Virtual-KITTI datasets among recent stereo-LiDAR fusion methods.