SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for RGB+Thermal 3D Reconstruction

TL;DR

SEAR is a fine-tuning strategy that adapts pretrained visual geometry transformers for effective RGB-thermal 3D reconstruction and pose estimation, outperforming state-of-the-art methods with minimal inference overhead.

Contribution

The paper introduces SEAR, a simple and efficient fine-tuning approach that enables pretrained geometry transformers to handle multimodal RGB-T inputs for 3D reconstruction.

Findings

Significant performance improvements over state-of-the-art methods (e.g., over 29% in AUC@30).

Effective multimodal pose estimation under challenging conditions like low light and smoke.

Introduction of a new RGB-thermal dataset for benchmarking multimodal 3D reconstruction.

Abstract

Foundational feed-forward visual geometry models enable accurate and efficient camera pose estimation and scene reconstruction by learning strong scene priors from massive RGB datasets. However, their effectiveness drops when applied to mixed sensing modalities, such as RGB-thermal (RGB-T) images. We observe that while a visual geometry grounded transformer pretrained on RGB data generalizes well to thermal-only reconstruction, it struggles to align RGB and thermal modalities when processed jointly. To address this, we propose SEAR, a simple yet efficient fine-tuning strategy that adapts a pretrained geometry transformer to multimodal RGB-T inputs. Despite being trained on a relatively small RGB-T dataset, our approach significantly outperforms state-of-the-art methods for 3D reconstruction and camera pose estimation, achieving significant improvements over all metrics (e.g., over 29\% in AUC@30) and delivering higher detail and consistency between modalities with negligible overhead in inference time compared to the original pretrained model. Notably, SEAR enables reliable multimodal pose estimation and reconstruction even under challenging conditions, such as low lighting and dense smoke. We validate our architecture through extensive ablation studies, demonstrating how the model aligns both modalities. Additionally, we introduce a new dataset featuring RGB and thermal sequences captured at different times, viewpoints, and illumination conditions, providing a robust benchmark for future work in multimodal 3D scene reconstruction. Code and models are publicly available at https://www.github.com/Schindler-EPFL-Lab/SEAR.