SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimator

TL;DR

SemCal introduces a neural mutual information estimator-based method for automatic, targetless LiDAR-camera calibration leveraging semantic data, optimized via a differentiable objective and initial registration, validated on synthetic and real-world datasets.

Contribution

It presents a novel semantic-level mutual information approach with a differentiable optimization framework for LiDAR-camera calibration, including an initial registration method.

Findings

Demonstrates robustness and accuracy on synthetic datasets.

Shows improved calibration results on KITTI360 and RELLIS-3D datasets.

Effectively uses semantic information and neural MI estimation for calibration.

Abstract

This paper proposes SemCal: an automatic, targetless, extrinsic calibration algorithm for a LiDAR and camera system using semantic information. We leverage a neural information estimator to estimate the mutual information (MI) of semantic information extracted from each sensor measurement, facilitating semantic-level data association. By using a matrix exponential formulation of the $se(3)$ transformation and a kernel-based sampling method to sample from camera measurement based on LiDAR projected points, we can formulate the LiDAR-Camera calibration problem as a novel differentiable objective function that supports gradient-based optimization methods. We also introduce a semantic-based initial calibration method using 2D MI-based image registration and Perspective-n-Point (PnP) solver. To evaluate performance, we demonstrate the robustness of our method and quantitatively analyze the accuracy using a synthetic dataset. We also evaluate our algorithm qualitatively on an urban dataset (KITTI360) and an off-road dataset (RELLIS-3D) benchmark datasets using both hand-annotated ground truth labels as well as labels predicted by the state-of-the-art deep learning models, showing improvement over recent comparable calibration approaches.