Hyperspectral vs. RGB for Pedestrian Segmentation in Urban Driving Scenes: A Comparative Study

TL;DR

This study compares hyperspectral imaging and RGB for pedestrian segmentation in urban driving, showing that optimal band selection in hyperspectral data improves segmentation accuracy over RGB in safety-critical automotive scenarios.

Contribution

It introduces a method for converting hyperspectral data into three channels using optimal band selection, enhancing pedestrian segmentation performance in automotive perception systems.

Findings

CSNR-JMIM outperforms RGB in IoU and F1-score for pedestrian segmentation

Hyperspectral data with optimal band selection reduces false positives

Spectral discrimination improves segmentation accuracy in urban scenes

Abstract

Pedestrian segmentation in automotive perception systems faces critical safety challenges due to metamerism in RGB imaging, where pedestrians and backgrounds appear visually indistinguishable.. This study investigates the potential of hyperspectral imaging (HSI) for enhanced pedestrian segmentation in urban driving scenarios using the Hyperspectral City v2 (H-City) dataset. We compared standard RGB against two dimensionality-reduction approaches by converting 128-channel HSI data into three-channel representations: Principal Component Analysis (PCA) and optimal band selection using Contrast Signal-to-Noise Ratio with Joint Mutual Information Maximization (CSNR-JMIM). Three semantic segmentation models were evaluated: U-Net, DeepLabV3+, and SegFormer. CSNR-JMIM consistently outperformed RGB with an average improvements of 1.44% in Intersection over Union (IoU) and 2.18% in F1-score for pedestrian segmentation. Rider segmentation showed similar gains with 1.43% IoU and 2.25% F1-score improvements. These improved performance results from enhanced spectral discrimination of optimally selected HSI bands effectively reducing false positives. This study demonstrates robust pedestrian segmentation through optimal HSI band selection, showing significant potential for safety-critical automotive applications.