Generative Model-Based Fusion for Improved Few-Shot Semantic Segmentation of Infrared Images

TL;DR

This paper introduces a generative model-based fusion approach to enhance few-shot semantic segmentation of infrared images, addressing data scarcity and modality differences without relying on paired RGB-IR data, and achieves superior results on IR datasets.

Contribution

The work proposes a novel fusion ensemble module and data synthesis strategies for IR FSS, eliminating the need for paired RGB images and improving segmentation performance.

Findings

Outperforms state-of-the-art IR FSS models on multiple datasets.

Effective data augmentation via IR data synthesis.

Enhanced support-query relationship modeling through fusion techniques.

Abstract

Infrared (IR) imaging is commonly used in various scenarios, including autonomous driving, fire safety and defense applications. Thus, semantic segmentation of such images is of great interest. However, this task faces several challenges, including data scarcity, differing contrast and input channel number compared to natural images, and emergence of classes not represented in databases in certain scenarios, such as defense applications. Few-shot segmentation (FSS) provides a framework to overcome these issues by segmenting query images using a few labeled support samples. However, existing FSS models for IR images require paired visible RGB images, which is a major limitation since acquiring such paired data is difficult or impossible in some applications. In this work, we develop new strategies for FSS of IR images by using generative modeling and fusion techniques. To this end, we propose to synthesize auxiliary data to provide additional channel information to complement the limited contrast in the IR images, as well as IR data synthesis for data augmentation. Here, the former helps the FSS model to better capture the relationship between the support and query sets, while the latter addresses the issue of data scarcity. Finally, to further improve the former aspect, we propose a novel fusion ensemble module for integrating the two different modalities. Our methods are evaluated on different IR datasets, and improve upon the state-of-the-art (SOTA) FSS models.