Learning Cross-Spectral Point Features with Task-Oriented Training

TL;DR

This paper introduces a novel method for learning cross-spectral point features that effectively integrate thermal and visible imagery for UAV navigation, especially in low-visibility conditions, by training on matching and registration tasks.

Contribution

The work proposes a new training approach for cross-spectral feature networks using matching and registration tasks, improving thermal-visible image integration for navigation.

Findings

Achieved below 10-pixel registration error in over 75% of cases on the MultiPoint dataset.

Model trained on matching tasks can be integrated into classical registration pipelines.

Demonstrated effectiveness in low-visibility environments for UAV navigation.

Abstract

Unmanned aerial vehicles (UAVs) enable operations in remote and hazardous environments, yet the visible-spectrum, camera-based navigation systems often relied upon by UAVs struggle in low-visibility conditions. Thermal cameras, which capture long-wave infrared radiation, are able to function effectively in darkness and smoke, where visible-light cameras fail. This work explores learned cross-spectral (thermal-visible) point features as a means to integrate thermal imagery into established camera-based navigation systems. Existing methods typically train a feature network's detection and description outputs directly, which often focuses training on image regions where thermal and visible-spectrum images exhibit similar appearance. Aiming to more fully utilize the available data, we propose a method to train the feature network on the tasks of matching and registration. We run our feature network on thermal-visible image pairs, then feed the network response into a differentiable registration pipeline. Losses are applied to the matching and registration estimates of this pipeline. Our selected model, trained on the task of matching, achieves a registration error (corner error) below 10 pixels for more than 75% of estimates on the MultiPoint dataset. We further demonstrate that our model can also be used with a classical pipeline for matching and registration.