Reliable Explainability of Deep Learning Spatial-Spectral Classifiers for Improved Semantic Segmentation in Autonomous Driving

TL;DR

This paper proposes a reliable explainability method for deep learning models used in hyperspectral image-based semantic segmentation for autonomous driving, addressing limitations of existing saliency techniques and improving model robustness.

Contribution

It introduces a new explainability approach leveraging activations and weights to better interpret spectral contributions in DNNs for semantic segmentation.

Findings

HSI-based models outperform 3-channel and single-channel DNNs in accuracy.

Normalization of spectral signatures enhances DNN robustness in real-world scenarios.

The proposed explainability method provides more reliable insights into spectral feature importance.

Abstract

Integrating hyperspectral imagery (HSI) with deep neural networks (DNNs) can strengthen the accuracy of intelligent vision systems by combining spectral and spatial information, which is useful for tasks like semantic segmentation in autonomous driving. To advance research in such safety-critical systems, determining the precise contribution of spectral information to complex DNNs' output is needed. To address this, several saliency methods, such as class activation maps (CAM), have been proposed primarily for image classification. However, recent studies have raised concerns regarding their reliability. In this paper, we address their limitations and propose an alternative approach by leveraging the data provided by activations and weights from relevant DNN layers to better capture the relationship between input features and predictions. The study aims to assess the superior performance of HSI compared to 3-channel and single-channel DNNs. We also address the influence of spectral signature normalization for enhancing DNN robustness in real-world driving conditions.