End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking

TL;DR

This paper introduces an end-to-end hyperspectral object tracking framework that jointly optimizes material decomposition and target localization, leveraging material prompts and spectral unmixing for improved robustness.

Contribution

It proposes a novel joint optimization approach coupling material unmixing with tracking, using a dual-branch wavelet-enhanced material prompt module for better material representation.

Findings

Achieves state-of-the-art results on hyperspectral tracking benchmarks.

Demonstrates the effectiveness of end-to-end joint optimization of material and localization.

Validates the generality of the framework across different unmixing backbones.

Abstract

Hyperspectral imagery encodes rich material properties that can improve tracking robustness under appearance ambiguity, illumination change, and background clutter. However, due to the limited availability of hyperspectral video data, many existing methods adapt pretrained RGB trackers via spatial or channel fusion strategies, largely neglecting the intrinsic material information in hyperspectral imagery. Moreover, the few material-aware approaches typically rely on external spectral unmixing pipelines that are decoupled from the tracking objective, limiting effective optimization of material representations for target localization. To address these limitations, we formulate hyperspectral object tracking as a joint optimization problem of material decomposition and target localization, coupling the two tasks via a weighted target-oriented unmixing loss that explicitly aligns material representations with localization accuracy. Specifically, we propose a material representation decomposition module for deep learning-based spectral unmixing with adaptive frequency decomposition. Building on the decomposed material representations, we further introduce a dual-branch wavelet-enhanced material prompt module that learns low- and high-frequency material prompts through efficient spatial-material interactions in the frequency domain. The framework is model-agnostic and can be seamlessly generalized to different unmixing backbones. Extensive experiments on standard hyperspectral tracking benchmarks demonstrate state-of-the-art performance and validate the effectiveness of the proposed end-to-end material-aware tracking framework. Code is available at https://github.com/han030927/E2EMPT.