Neural Real-Time Recalibration for Infrared Multi-Camera Systems

TL;DR

This paper introduces a neural network-based method for real-time, highly-accurate calibration of infrared multi-camera systems, overcoming limitations of traditional techniques by enabling dynamic on-the-fly recalibration with robustness to perturbations.

Contribution

The paper presents a novel neural recalibration approach that integrates a differentiable projection model and dynamic pose synthesis for real-time multi-camera infrared system calibration.

Findings

Outperforms traditional calibration methods in accuracy.

Operates in real-time with high robustness to perturbations.

Applicable to both onboard processing and image-based systems.

Abstract

Currently, there are no learning-free or neural techniques for real-time recalibration of infrared multi-camera systems. In this paper, we address the challenge of real-time, highly-accurate calibration of multi-camera infrared systems, a critical task for time-sensitive applications. Unlike traditional calibration techniques that lack adaptability and struggle with on-the-fly recalibrations, we propose a neural network-based method capable of dynamic real-time calibration. The proposed method integrates a differentiable projection model that directly correlates 3D geometries with their 2D image projections and facilitates the direct optimization of both intrinsic and extrinsic camera parameters. Key to our approach is the dynamic camera pose synthesis with perturbations in camera parameters, emulating realistic operational challenges to enhance model robustness. We introduce two model variants: one designed for multi-camera systems with onboard processing of 2D points, utilizing the direct 2D projections of 3D fiducials, and another for image-based systems, employing color-coded projected points for implicitly establishing correspondence. Through rigorous experimentation, we demonstrate our method is more accurate than traditional calibration techniques with or without perturbations while also being real-time, marking a significant leap in the field of real-time multi-camera system calibration. The source code can be found at https://github.com/theICTlab/neural-recalibration