INF: Implicit Neural Fusion for LiDAR and Camera

TL;DR

This paper introduces Implicit Neural Fusion (INF), a novel approach that unifies LiDAR and camera data through neural density and color fields, improving sensor fusion accuracy without manual calibration.

Contribution

INF leverages implicit neural representations to fuse LiDAR and camera data, simultaneously estimating poses and extrinsics, reducing calibration complexity.

Findings

High accuracy in sensor fusion demonstrated

Stable performance across different scenarios

Automatic pose and extrinsic parameter estimation

Abstract

Sensor fusion has become a popular topic in robotics. However, conventional fusion methods encounter many difficulties, such as data representation differences, sensor variations, and extrinsic calibration. For example, the calibration methods used for LiDAR-camera fusion often require manual operation and auxiliary calibration targets. Implicit neural representations (INRs) have been developed for 3D scenes, and the volume density distribution involved in an INR unifies the scene information obtained by different types of sensors. Therefore, we propose implicit neural fusion (INF) for LiDAR and camera. INF first trains a neural density field of the target scene using LiDAR frames. Then, a separate neural color field is trained using camera images and the trained neural density field. Along with the training process, INF both estimates LiDAR poses and optimizes extrinsic parameters. Our experiments demonstrate the high accuracy and stable performance of the proposed method.