Validation & Exploration of Multimodal Deep-Learning Camera-Lidar Calibration models

TL;DR

This paper evaluates deep learning models for real-time calibration of LiDAR and camera sensors, comparing open-source architectures to identify the most accurate and reliable approach for sensor fusion.

Contribution

It introduces a comprehensive comparison of CNN-based calibration models, highlighting their performance and limitations in dynamic multi-modal sensor alignment.

Findings

LCCNet outperforms other models in accuracy

Open-source models require fine-tuning for optimal results

Identifies areas for improvement in deep learning calibration methods

Abstract

This article presents an innovative study in exploring, evaluating, and implementing deep learning architectures for the calibration of multi-modal sensor systems. The focus behind this is to leverage the use of sensor fusion to achieve dynamic, real-time alignment between 3D LiDAR and 2D Camera sensors. static calibration methods are tedious and time-consuming, which is why we propose utilizing Conventional Neural Networks (CNN) coupled with geometrically informed learning to solve this issue. We leverage the foundational principles of Extrinsic LiDAR-Camera Calibration tools such as RegNet, CalibNet, and LCCNet by exploring open-source models that are available online and comparing our results with their corresponding research papers. Requirements for extracting these visual and measurable outputs involved tweaking source code, fine-tuning, training, validation, and testing for each of these frameworks for equal comparisons. This approach aims to investigate which of these advanced networks produces the most accurate and consistent predictions. Through a series of experiments, we reveal some of their shortcomings and areas for potential improvements along the way. We find that LCCNet yields the best results out of all the models that we validated.