SSFT: A Lightweight Spectral-Spatial Fusion Transformer for Generic Hyperspectral Classification

TL;DR

The paper introduces SSFT, a lightweight spectral-spatial transformer for hyperspectral classification, achieving state-of-the-art results with fewer parameters across diverse datasets.

Contribution

It proposes a novel spectral-spatial fusion transformer that is compact, effective, and adaptable to various hyperspectral classification tasks.

Findings

SSFT ranks first on the HSI-Benchmark with less than 2% of previous model parameters.

SSFT remains competitive on the larger SpectralEarth benchmark despite its small size.

Both spectral and spatial pathways are essential, with spatial modeling being most impactful.

Abstract

Hyperspectral imaging enables fine-grained recognition of materials by capturing rich spectral signatures, but learning robust classifiers is challenging due to high dimensionality, spectral redundancy, limited labeled data, and strong domain shifts. Beyond earth observation, labeled HSI data is often scarce and imbalanced, motivating compact models for generic hyperspectral classification across diverse acquisition regimes. We propose the lightweight Spectral-Spatial Fusion Transformer (SSFT), which factorizes representation learning into spectral and spatial pathways and integrates them via cross-attention to capture complementary wavelength-dependent and structural information. We evaluate our SSFT on the challenging HSI-Benchmark, a heterogeneous multi-dataset benchmark covering earth observation, fruit condition assessment, and fine-grained material recognition. SSFT achieves state-of-the-art overall performance, ranking first while using less than 2% of the parameters of the previous leading method. We further evaluate transfer to the substantially larger SpectralEarth benchmark under the official protocol, where SSFT remains competitive despite its compact size. Ablation studies show that both spectral and spatial pathways are crucial, with spatial modeling contributing most, and that SSFT remains robust without data augmentation.