Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning

TL;DR

This paper introduces a holistic, targetless calibration approach for camera, lidar, and radar sensors in autonomous vehicles, utilizing physics, semantics, and self-supervised learning to improve robustness and reduce manual effort.

Contribution

It proposes two novel targetless calibration methods that leverage physical and semantic information, enabling online and self-supervised calibration of multi-sensor systems.

Findings

Effective joint calibration of camera, lidar, and radar achieved

Methods operate online without known targets

Improved robustness and reduced manual calibration effort

Abstract

Achieving safe and reliable autonomous driving relies greatly on the ability to achieve an accurate and robust perception system; however, this cannot be fully realized without precisely calibrated sensors. Environmental and operational conditions as well as improper maintenance can produce calibration errors inhibiting sensor fusion and, consequently, degrading the perception performance. Traditionally, sensor calibration is performed in a controlled environment with one or more known targets. Such a procedure can only be carried out in between drives and requires manual operation; a tedious task if needed to be conducted on a regular basis. This sparked a recent interest in online targetless methods, capable of yielding a set of geometric transformations based on perceived environmental features, however, the required redundancy in sensing modalities makes this task even more challenging, as the features captured by each modality and their distinctiveness may vary. We present a holistic approach to performing joint calibration of a camera-lidar-radar trio. Leveraging prior knowledge and physical properties of these sensing modalities together with semantic information, we propose two targetless calibration methods within a cost minimization framework once via direct online optimization, and second via self-supervised learning (SSL).