Uncertainty-aware Spatial-Frequency Registration and Fusion for Infrared and Visible Images

TL;DR

This paper introduces a novel spatial-frequency registration and fusion framework for infrared and visible images that effectively handles misalignments by incorporating uncertainty estimation and frequency domain consistency.

Contribution

It proposes a unified pipeline with iterative registration, uncertainty modeling, and frequency-domain supervision to improve robustness and accuracy in image fusion under unregistered conditions.

Findings

Achieves superior registration accuracy across datasets.

Effectively reduces error accumulation in multi-scale registration.

Produces high-quality fused images with enhanced visual clarity.

Abstract

Infrared and Visible Image Fusion (IVIF) has shown promise in visual tasks under challenging environments, but fusion under unregistered conditions faces inherent misalignments. Current studies to solve them either predict the deformation parameters coarse-to-fine (i.e., coarse registration and fine registration) or estimate the deformation fields in multi-scales for registration. Though straightforward, they overlook the cumulative errors in registration, which contaminate the fusion stage and severely deteriorate the resulting images. We introduce the Spatial-Frequency Registration and Fusion (SFRF) framework, which incorporates uncertainty estimation and infrared thermal radiation distribution consistency into a unified pipeline to handle the error accumulation for robust registration and fusion across both spatial and frequency domains. Specifically, SFRF constructs a Multi-scale Iterative Registration (MIR) framework that iteratively refines the deformation field across scales, leveraging uncertainty estimation at each stage to mitigate error accumulation and enhance alignment accuracy dynamically. To ensure the accurate alignment of infrared thermal distributions during registration, thermal radiation distribution consistency is employed as a frequency-domain supervisory signal, promoting global consistency in the frequency domain. Based on the spatial-frequency alignment, SFRF further adopts a Dual-branch Spatial-Frequency Fusion (DSFF) module, which incorporates spatial geometric features and frequency distribution information to reconstruct visually appealing images. SFRF achieves impressive performance across diverse datasets.