SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation

TL;DR

SpecTr introduces a spectral transformer model that effectively captures long-range spectral dependencies and spatial-spectral features in hyperspectral pathology images, significantly improving segmentation accuracy without pre-training.

Contribution

The paper presents the first application of transformers for spectral context modeling in hyperspectral pathology image segmentation, incorporating sparsity and spectral normalization strategies.

Findings

Outperforms existing methods on a hyperspectral pathology segmentation benchmark.

Does not require pre-training, simplifying the training process.

Effectively models long-range spectral dependencies and spatial-spectral features.

Abstract

Hyperspectral imaging (HSI) unlocks the huge potential to a wide variety of applications relied on high-precision pathology image segmentation, such as computational pathology and precision medicine. Since hyperspectral pathology images benefit from the rich and detailed spectral information even beyond the visible spectrum, the key to achieve high-precision hyperspectral pathology image segmentation is to felicitously model the context along high-dimensional spectral bands. Inspired by the strong context modeling ability of transformers, we hereby, for the first time, formulate the contextual feature learning across spectral bands for hyperspectral pathology image segmentation as a sequence-to-sequence prediction procedure by transformers. To assist spectral context learning procedure, we introduce two important strategies: (1) a sparsity scheme enforces the learned contextual relationship to be sparse, so as to eliminates the distraction from the redundant bands; (2) a spectral normalization, a separate group normalization for each spectral band, mitigates the nuisance caused by heterogeneous underlying distributions of bands. We name our method Spectral Transformer (SpecTr), which enjoys two benefits: (1) it has a strong ability to model long-range dependency among spectral bands, and (2) it jointly explores the spatial-spectral features of HSI. Experiments show that SpecTr outperforms other competing methods in a hyperspectral pathology image segmentation benchmark without the need of pre-training. Code is available at https://github.com/hfut-xc-yun/SpecTr.