PAN-Crafter: Learning Modality-Consistent Alignment for PAN-Sharpening

TL;DR

PAN-Crafter introduces a novel framework for PAN-sharpening that explicitly addresses cross-modality misalignment using modality-consistent alignment, resulting in improved spectral fidelity and structural detail in high-resolution multi-spectral images.

Contribution

It proposes a new alignment framework with Modality-Adaptive Reconstruction and Cross-Modality Alignment-Aware Attention to mitigate misalignment effects in PAN-sharpening.

Findings

Outperforms state-of-the-art methods across multiple benchmarks.

Achieves 50.11× faster inference and uses less memory.

Demonstrates strong generalization on unseen datasets.

Abstract

PAN-sharpening aims to fuse high-resolution panchromatic (PAN) images with low-resolution multi-spectral (MS) images to generate high-resolution multi-spectral (HRMS) outputs. However, cross-modality misalignment -- caused by sensor placement, acquisition timing, and resolution disparity -- induces a fundamental challenge. Conventional deep learning methods assume perfect pixel-wise alignment and rely on per-pixel reconstruction losses, leading to spectral distortion, double edges, and blurring when misalignment is present. To address this, we propose PAN-Crafter, a modality-consistent alignment framework that explicitly mitigates the misalignment gap between PAN and MS modalities. At its core, Modality-Adaptive Reconstruction (MARs) enables a single network to jointly reconstruct HRMS and PAN images, leveraging PAN's high-frequency details as auxiliary self-supervision. Additionally, we introduce Cross-Modality Alignment-Aware Attention (CM3A), a novel mechanism that bidirectionally aligns MS texture to PAN structure and vice versa, enabling adaptive feature refinement across modalities. Extensive experiments on multiple benchmark datasets demonstrate that our PAN-Crafter outperforms the most recent state-of-the-art method in all metrics, even with 50.11$\times$ faster inference time and 0.63$\times$ the memory size. Furthermore, it demonstrates strong generalization performance on unseen satellite datasets, showing its robustness across different conditions.