DiffFormer: a Differential Spatial-Spectral Transformer for Hyperspectral Image Classification

TL;DR

DiffFormer introduces a novel differential spatial-spectral transformer architecture that significantly improves hyperspectral image classification accuracy and efficiency by emphasizing subtle spectral-spatial variations.

Contribution

The paper proposes DiffFormer, a new transformer-based framework with differential multi-head self-attention and 3D convolutional tokenization for hyperspectral image classification, addressing spectral redundancy and spatial discontinuity.

Findings

Outperforms existing SOTA methods in accuracy

Demonstrates high computational efficiency and scalability

Provides detailed analysis of model complexity

Abstract

Hyperspectral image classification (HSIC) has gained significant attention because of its potential in analyzing high-dimensional data with rich spectral and spatial information. In this work, we propose the Differential Spatial-Spectral Transformer (DiffFormer), a novel framework designed to address the inherent challenges of HSIC, such as spectral redundancy and spatial discontinuity. The DiffFormer leverages a Differential Multi-Head Self-Attention (DMHSA) mechanism, which enhances local feature discrimination by introducing differential attention to accentuate subtle variations across neighboring spectral-spatial patches. The architecture integrates Spectral-Spatial Tokenization through three-dimensional (3D) convolution-based patch embeddings, positional encoding, and a stack of transformer layers equipped with the SWiGLU activation function for efficient feature extraction (SwiGLU is a variant of the Gated Linear Unit (GLU) activation function). A token-based classification head further ensures robust representation learning, enabling precise labeling of hyperspectral pixels. Extensive experiments on benchmark hyperspectral datasets demonstrate the superiority of DiffFormer in terms of classification accuracy, computational efficiency, and generalizability, compared to existing state-of-the-art (SOTA) methods. In addition, this work provides a detailed analysis of computational complexity, showcasing the scalability of the model for large-scale remote sensing applications. The source code will be made available at \url{https://github.com/mahmad000/DiffFormer} after the first round of revision.