What Really Matters for Learning-based LiDAR-Camera Calibration

TL;DR

This paper critically analyzes learning-based LiDAR-camera calibration methods, revealing they often function as retrieval networks focusing on single-modality data rather than true cross-modality correspondences, and discusses how data formats influence performance.

Contribution

It systematically examines existing methods, identifies their limitations, and provides insights into the underlying principles to guide future research in practical calibration applications.

Findings

Most methods operate as retrieval networks rather than true cross-modality matching.

Input data format and preprocessing significantly affect calibration performance.

Regression-based methods have critical limitations in complex real-world scenarios.

Abstract

Calibration is an essential prerequisite for the accurate data fusion of LiDAR and camera sensors. Traditional calibration techniques often require specific targets or suitable scenes to obtain reliable 2D-3D correspondences. To tackle the challenge of target-less and online calibration, deep neural networks have been introduced to solve the problem in a data-driven manner. While previous learning-based methods have achieved impressive performance on specific datasets, they still struggle in complex real-world scenarios. Most existing works focus on improving calibration accuracy but overlook the underlying mechanisms. In this paper, we revisit the development of learning-based LiDAR-Camera calibration and encourage the community to pay more attention to the underlying principles to advance practical applications. We systematically analyze the paradigm of mainstream learning-based methods, and identify the critical limitations of regression-based methods with the widely used data generation pipeline. Our findings reveal that most learning-based methods inadvertently operate as retrieval networks, focusing more on single-modality distributions rather than cross-modality correspondences. We also investigate how the input data format and preprocessing operations impact network performance and summarize the regression clues to inform further improvements.