AnyThermal: Towards Learning Universal Representations for Thermal Perception

TL;DR

AnyThermal is a universal thermal feature extractor that leverages foundation models and diverse data to perform well across multiple tasks and environments without task-specific training.

Contribution

It introduces a thermal backbone distilled from visual foundation models and the TartanRGBT dataset for diverse environment training.

Findings

Achieves state-of-the-art results across various tasks and environments.

Improves performance by up to 36% on existing datasets.

Demonstrates robustness and versatility of the thermal backbone.

Abstract

We present AnyThermal, a thermal backbone that captures robust task-agnostic thermal features suitable for a variety of tasks such as cross-modal place recognition, thermal segmentation, and monocular depth estimation using thermal images. Existing thermal backbones that follow task-specific training from small-scale data result in utility limited to a specific environment and task. Unlike prior methods, AnyThermal can be used for a wide range of environments (indoor, aerial, off-road, urban) and tasks, all without task-specific training. Our key insight is to distill the feature representations from visual foundation models such as DINOv2 into a thermal encoder using thermal data from these multiple environments. To bridge the diversity gap of the existing RGB-Thermal datasets, we introduce the TartanRGBT platform, the first open-source data collection platform with synced RGB-Thermal image acquisition. We use this payload to collect the TartanRGBT dataset - a diverse and balanced dataset collected in 4 environments. We demonstrate the efficacy of AnyThermal and TartanRGBT, achieving state-of-the-art results with improvements of up to 36% across diverse environments and downstream tasks on existing datasets.