Frequency-Adaptive Pan-Sharpening with Mixture of Experts

TL;DR

This paper introduces FAME, a frequency-adaptive mixture of experts framework for pan-sharpening that effectively reconstructs high-frequency details in multi-spectral images by leveraging frequency domain separation and dynamic expert weighting.

Contribution

The paper proposes a novel frequency domain-based pan-sharpening framework with adaptive frequency separation and expert mixture modules, improving reconstruction accuracy and generalization.

Findings

Outperforms state-of-the-art methods on multiple datasets

Demonstrates strong generalization to real-world scenes

Effectively reconstructs high-frequency details in multi-spectral images

Abstract

Pan-sharpening involves reconstructing missing high-frequency information in multi-spectral images with low spatial resolution, using a higher-resolution panchromatic image as guidance. Although the inborn connection with frequency domain, existing pan-sharpening research has not almost investigated the potential solution upon frequency domain. To this end, we propose a novel Frequency Adaptive Mixture of Experts (FAME) learning framework for pan-sharpening, which consists of three key components: the Adaptive Frequency Separation Prediction Module, the Sub-Frequency Learning Expert Module, and the Expert Mixture Module. In detail, the first leverages the discrete cosine transform to perform frequency separation by predicting the frequency mask. On the basis of generated mask, the second with low-frequency MOE and high-frequency MOE takes account for enabling the effective low-frequency and high-frequency information reconstruction. Followed by, the final fusion module dynamically weights high-frequency and low-frequency MOE knowledge to adapt to remote sensing images with significant content variations. Quantitative and qualitative experiments over multiple datasets demonstrate that our method performs the best against other state-of-the-art ones and comprises a strong generalization ability for real-world scenes. Code will be made publicly at \url{https://github.com/alexhe101/FAME-Net}.