HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model

TL;DR

HyperSIGMA is a scalable, transformer-based foundation model for hyperspectral image interpretation that unifies multiple tasks and scenes, leveraging a novel sparse sampling attention mechanism and a large-scale dataset for superior performance.

Contribution

The paper introduces HyperSIGMA, a vision transformer model with a novel SSA mechanism and a large hyperspectral dataset, enabling cross-task and cross-scene HSI interpretation.

Findings

HyperSIGMA outperforms state-of-the-art methods across various HSI tasks.

The SSA mechanism effectively captures diverse contextual features.

HyperSIGMA demonstrates high scalability, robustness, and real-world applicability.

Abstract

Accurate hyperspectral image (HSI) interpretation is critical for providing valuable insights into various earth observation-related applications such as urban planning, precision agriculture, and environmental monitoring. However, existing HSI processing methods are predominantly task-specific and scene-dependent, which severely limits their ability to transfer knowledge across tasks and scenes, thereby reducing the practicality in real-world applications. To address these challenges, we present HyperSIGMA, a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes, scalable to over one billion parameters. To overcome the spectral and spatial redundancy inherent in HSIs, we introduce a novel sparse sampling attention (SSA) mechanism, which effectively promotes the learning of diverse contextual features and serves as the basic block of HyperSIGMA. HyperSIGMA integrates spatial and spectral features using a specially designed spectral enhancement module. In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450K hyperspectral images, significantly surpassing existing datasets in scale. Extensive experiments on various high-level and low-level HSI tasks demonstrate HyperSIGMA's versatility and superior representational capability compared to current state-of-the-art methods. Moreover, HyperSIGMA shows significant advantages in scalability, robustness, cross-modal transferring capability, real-world applicability, and computational efficiency. The code and models will be released at https://github.com/WHU-Sigma/HyperSIGMA.