Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared

TL;DR

This paper introduces a novel dictionary-guided framework for infrared-visible image fusion that effectively handles missing IR data by operating in the coefficient domain, improving perceptual quality and detection performance.

Contribution

It proposes a shared convolutional dictionary and a coefficient-domain inference-fusion pipeline, a first approach to address missing-IR fusion with interpretability and prior preservation.

Findings

Consistent improvements in perceptual quality under missing-IR conditions.

Enhanced downstream detection performance with the proposed method.

First framework to jointly learn a shared dictionary and perform coefficient-domain inference for missing-IR fusion.

Abstract

Infrared-visible (IR-VIS) image fusion is vital for perception and security, yet most methods rely on the availability of both modalities during training and inference. When the infrared modality is absent, pixel-space generative substitutes become hard to control and inherently lack interpretability. We address missing-IR fusion by proposing a dictionary-guided, coefficient-domain framework built upon a shared convolutional dictionary. The pipeline comprises three key components: (1) Joint Shared-dictionary Representation Learning (JSRL) learns a unified and interpretable atom space shared by both IR and VIS modalities; (2) VIS-Guided IR Inference (VGII) transfers VIS coefficients to pseudo-IR coefficients in the coefficient domain and performs a one-step closed-loop refinement guided by a frozen large language model as a weak semantic prior; and (3) Adaptive Fusion via Representation Inference (AFRI) merges VIS structures and inferred IR cues at the atom level through window attention and convolutional mixing, followed by reconstruction with the shared dictionary. This encode-transfer-fuse-reconstruct pipeline avoids uncontrolled pixel-space generation while ensuring prior preservation within interpretable dictionary-coefficient representation. Experiments under missing-IR settings demonstrate consistent improvements in perceptual quality and downstream detection performance. To our knowledge, this represents the first framework that jointly learns a shared dictionary and performs coefficient-domain inference-fusion to tackle missing-IR fusion. The source code is publicly available at https://github.com/harukiv/DCMIF.