Hyperspectral Smoke Segmentation via Mixture of Prototypes

TL;DR

This paper introduces a novel hyperspectral smoke segmentation method using a mixture of prototypes network, addressing spectral variability and demonstrating superior performance on a new dataset for wildfire and industrial safety applications.

Contribution

It presents the first hyperspectral smoke segmentation dataset and a mixture of prototypes network with adaptive band weighting for improved segmentation accuracy.

Findings

Superior performance on hyperspectral data

Effective adaptive spectral band weighting

Validated on a newly collected real-world dataset

Abstract

Smoke segmentation is critical for wildfire management and industrial safety applications. Traditional visible-light-based methods face limitations due to insufficient spectral information, particularly struggling with cloud interference and semi-transparent smoke regions. To address these challenges, we introduce hyperspectral imaging for smoke segmentation and present the first hyperspectral smoke segmentation dataset (HSSDataset) with carefully annotated samples collected from over 18,000 frames across 20 real-world scenarios using a Many-to-One annotations protocol. However, different spectral bands exhibit varying discriminative capabilities across spatial regions, necessitating adaptive band weighting strategies. We decompose this into three technical challenges: spectral interaction contamination, limited spectral pattern modeling, and complex weighting router problems. We propose a mixture of prototypes (MoP) network with: (1) Band split for spectral isolation, (2) Prototype-based spectral representation for diverse patterns, and (3) Dual-level router for adaptive spatial-aware band weighting. We further construct a multispectral dataset (MSSDataset) with RGB-infrared images. Extensive experiments validate superior performance across both hyperspectral and multispectral modalities, establishing a new paradigm for spectral-based smoke segmentation.