AWM-Fuse: Multi-Modality Image Fusion for Adverse Weather via Global and Local Text Perception

TL;DR

AWM-Fuse introduces a multi-modality image fusion approach that utilizes global and local textual perception to enhance scene clarity and semantic understanding in adverse weather conditions, outperforming existing methods.

Contribution

The paper presents a novel fusion method integrating global and local text perception modules, leveraging BLIP and ChatGPT for improved semantic-aware image fusion under adverse weather.

Findings

Outperforms state-of-the-art in adverse weather scenarios

Enhances semantic perception through textual guidance

Improves downstream task performance

Abstract

Multi-modality image fusion (MMIF) in adverse weather aims to address the loss of visual information caused by weather-related degradations, providing clearer scene representations. Although less studies have attempted to incorporate textual information to improve semantic perception, they often lack effective categorization and thorough analysis of textual content. In response, we propose AWM-Fuse, a novel fusion method for adverse weather conditions, designed to handle multiple degradations through global and local text perception within a unified, shared weight architecture. In particular, a global feature perception module leverages BLIP-produced captions to extract overall scene features and identify primary degradation types, thus promoting generalization across various adverse weather conditions. Complementing this, the local module employs detailed scene descriptions produced by ChatGPT to concentrate on specific degradation effects through concrete textual cues, thereby capturing finer details. Furthermore, textual descriptions are used to constrain the generation of fusion images, effectively steering the network learning process toward better alignment with real semantic labels, thereby promoting the learning of more meaningful visual features. Extensive experiments demonstrate that AWM-Fuse outperforms current state-of-the-art methods in complex weather conditions and downstream tasks. Our code is available at https://github.com/Feecuin/AWM-Fuse.