PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation

TL;DR

This paper introduces PAIF, a perception-aware infrared-visible image fusion framework designed to enhance the robustness of semantic segmentation against adversarial attacks by balancing accuracy and defense strategies.

Contribution

It proposes a novel architecture search and adaptive learning strategy to improve segmentation robustness in adversarial scenes, addressing vulnerabilities of existing fusion methods.

Findings

Achieves 15.3% higher mIOU in adversarial scenes

Enhances robustness of semantic segmentation under attacks

Balances accuracy and robustness through architecture and learning strategies

Abstract

Infrared and visible image fusion is a powerful technique that combines complementary information from different modalities for downstream semantic perception tasks. Existing learning-based methods show remarkable performance, but are suffering from the inherent vulnerability of adversarial attacks, causing a significant decrease in accuracy. In this work, a perception-aware fusion framework is proposed to promote segmentation robustness in adversarial scenes. We first conduct systematic analyses about the components of image fusion, investigating the correlation with segmentation robustness under adversarial perturbations. Based on these analyses, we propose a harmonized architecture search with a decomposition-based structure to balance standard accuracy and robustness. We also propose an adaptive learning strategy to improve the parameter robustness of image fusion, which can learn effective feature extraction under diverse adversarial perturbations. Thus, the goals of image fusion (\textit{i.e.,} extracting complementary features from source modalities and defending attack) can be realized from the perspectives of architectural and learning strategies. Extensive experimental results demonstrate that our scheme substantially enhances the robustness, with gains of 15.3% mIOU of segmentation in the adversarial scene, compared with advanced competitors. The source codes are available at https://github.com/LiuZhu-CV/PAIF.