Spectral Gaps and Spatial Priors: Studying Hyperspectral Downstream Adaptation Using TerraMind

TL;DR

This paper explores how TerraMind, a multimodal geospatial foundation model, can adapt to hyperspectral imaging tasks without HSI-specific pretraining, highlighting the importance of native spectral support for improved performance.

Contribution

The study introduces and compares channel adaptation strategies for TerraMind to handle hyperspectral data, establishing a baseline for future HSI integration in multimodal models.

Findings

Deep learning models with native HSI support perform better.

TerraMind can adapt to HSI tasks via band selection with moderate performance loss.

Native spectral tokenization is crucial for future multimodal architectures.

Abstract

Geospatial Foundation Models (GFMs) typically lack native support for Hyperspectral Imaging (HSI) due to the complexity and sheer size of high-dimensional spectral data. This study investigates the adaptability of TerraMind, a multimodal GFM, to address HSI downstream tasks \emph{without} HSI-specific pretraining. Therefore, we implement and compare two channel adaptation strategies: Naive Band Selection and physics-aware Spectral Response Function (SRF) grouping. Overall, our results indicate a general superiority of deep learning models with native support of HSI data. Our experiments also demonstrate the ability of TerraMind to adapt to HSI downstream tasks through band selection with moderate performance decline. Therefore, the findings of this research establish a critical baseline for HSI integration, motivating the need for native spectral tokenization in future multimodal model architectures.