Spectral-Enhanced Transformers: Leveraging Large-Scale Pretrained Models for Hyperspectral Object Tracking

TL;DR

This paper introduces a novel method that adapts large pretrained transformer models for hyperspectral object tracking by using a learnable spatial-spectral fusion module and a cross-modality training pipeline, achieving high performance with minimal training.

Contribution

It presents an adaptive spatial-spectral token fusion module and a cross-modality training pipeline for hyperspectral tracking using pretrained transformers, addressing data scarcity and training efficiency.

Findings

Superior tracking performance with fewer training iterations

Effective learning across different sensor modalities

Enhanced spectral and spatial feature extraction

Abstract

Hyperspectral object tracking using snapshot mosaic cameras is emerging as it provides enhanced spectral information alongside spatial data, contributing to a more comprehensive understanding of material properties. Using transformers, which have consistently outperformed convolutional neural networks (CNNs) in learning better feature representations, would be expected to be effective for Hyperspectral object tracking. However, training large transformers necessitates extensive datasets and prolonged training periods. This is particularly critical for complex tasks like object tracking, and the scarcity of large datasets in the hyperspectral domain acts as a bottleneck in achieving the full potential of powerful transformer models. This paper proposes an effective methodology that adapts large pretrained transformer-based foundation models for hyperspectral object tracking. We propose an adaptive, learnable spatial-spectral token fusion module that can be extended to any transformer-based backbone for learning inherent spatial-spectral features in hyperspectral data. Furthermore, our model incorporates a cross-modality training pipeline that facilitates effective learning across hyperspectral datasets collected with different sensor modalities. This enables the extraction of complementary knowledge from additional modalities, whether or not they are present during testing. Our proposed model also achieves superior performance with minimal training iterations.