FIAN: A frequency information-adaptive network for spatial-frequency domain pansharpening
Yang Liu, Wei Wang, Weihe Li
TL;DR
This paper introduces FIAN, a new neural network for pansharpening that better preserves high-frequency details by adapting to frequency information.
Contribution
The novel frequency information-adaptive filter module and fusion strategy improve high-frequency detail preservation in pansharpening.
Findings
FIAN outperforms existing methods in preserving high-frequency details in pansharpened images.
The proposed frequency feature selection strategy enhances the network's representational power.
Extensive experiments on benchmark datasets confirm FIAN's competitive performance.
Abstract
Pansharpening aims to combine the spatial information from high-resolution panchromatic (PAN) images with the spectral information from low-resolution multispectral (LRMS) images generating high-resolution multispectral (HRMS) images. While Convolutional Neural Networks (CNNs) have shown impressive performance in pansharpening tasks, their tendency to focus more on low-frequency information will lead to suboptimal preservation of high-frequency details, which are crucial for producing HRMS images. Recent studies have highlighted the significance of frequency domain information in pansharpening, but existing methods often consider the network as a whole, overlooking the unique abilities of different layers in capturing high-frequency components. This oversight can result in the loss of fine details and limit the overall performance of pansharpening. To overcome these limitations, we…
Genes, proteins, chemicals, diseases, species, mutations and cell lines named across the full text — each resolved to its canonical identifier and authoritative record.
Click any figure to enlarge with its caption.
Fig 1
Fig 2
Fig 3
Fig 4
Fig 5
Fig 6
Fig 7
Fig 8
Fig 9
Fig 10
Fig 11- —Central science and technology development project led by Hubei Province
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Image Fusion Techniques · Image and Signal Denoising Methods · Remote-Sensing Image Classification
Introduction
High-resolution multispectral (HRMS) images are vital across a range of sectors, including agriculture, industry, and civil infrastructure. Due to the limitations of hardware equipment, HRMS cannot be directly acquired. Pansharpening overcomes this obstacle by effectively fusing high-resolution panchromatic (PAN) images with low-resolution multispectral(LRMS) images, thus synthesizing HRMS images [8, 76], providing better data support for the subsequent analysis and application of remote sensing images [2, 7, 12, 14, 20, 63].
Masi et al. [26] developed and trained PNN, the pioneering pansharpening network based on convolutional neural networks (CNNs), which employs a three-layer convolution structure. The success of PNN has spurred extensive research into CNN-based pansharpening methods, leading to substantial advancements in the field [3, 9, 29, 31, 33, 34, 37, 40–42, 45, 46].
Further investigation into the characteristics of CNN layers reveals notable frequency-related properties. As shown in Fig 1(a), we perform a frequency analysis on the features extracted by each layer of a representative nine-layer CNN at various stages of training. This study employs the discrete gradient as a measure to assess the frequency composition of layer features; larger gradient values suggest an increased proportion of high-frequency components.
Gradients of network features.Distribution of discrete gradients of network features at different training stages. (a): The evolution of discrete gradient values for each layer in the network as training iterations (epochs). (b): The changes in discrete gradient values for network layers with skip connections throughout the training iterations (epochs).
Fig 1(a) shows that during the initial phase of training (e.g., epoch=1), the gradients of shallow layer features substantially exceed those of deeper layers, indicating that the CNN primarily captures low-frequency image content. As training progresses (for example, epoch = 350), the gradients of the features of the deep layer gradually increase, eventually exceeding those of the shallower layers. This suggests that CNN begins to capture high-frequency details. By Epoch 1000, the gradients of deep layer features significantly outweigh those of shallow layers, implying that deeper layers predominantly focus on high-frequency image information. This observation highlights the frequency-selective nature of CNNs, exhibiting a transition from low-frequency to high-frequency feature extraction. Consequently, this frequency bias impacts the performance of pansharpening in two ways: firstly, it leads to a slower convergence rate of CNNs in the pansharpening task; secondly, the emphasis on low-frequency information in shallow layers may result in the loss of crucial high-frequency details during feature extraction.
To mitigate these challenges, researchers have explored the incorporation of skip connections, such as in DiCNN [50]. Fig 1(b) depicts the distribution characteristics of the finite difference gradients across CNN layers after introducing skip connections. It is evident that odd and even layers exhibit distinct gradient distribution patterns, collectively forming a “W" shape. This suggests that skip connections modulate the frequency extraction behavior of CNNs to a certain degree. By propagating the output features of the current layer along with those of the previous layer to the subsequent layer, skip connections help alleviate the loss of high-frequency information that arises from the limited high-frequency extraction capability of the current layer. However, closer examination reveals that within odd and even layer groups, the frequency extraction characteristics still align with those observed in Fig 1(a). This implies that while skip connections provide some relief for high-frequency loss, they do not fundamentally resolve the frequency extraction bias inherent to CNNs [22].
In an effort to further enhance pansharpening performance, researchers have begun to investigate methods that synergize frequency and spatial domain information. However, existing approaches often treat CNNs as monolithic entities, disregarding the varying capacities of different network layers in extracting high-frequency information. Naively fusing frequency domain information with spatial domain features may fail to provide the most appropriate frequency components for the current CNN layer, exacerbating the loss of high-frequency details and compromising the effectiveness of dual-domain information fusion.
To tackle these challenges, we introduce FIAN, an innovative frequency information-adaptive network designed specifically for spatial-frequency domain pansharpening. At the heart of FIAN lies the frequency information-adaptive filter module, which intelligently extracts vital frequency-domain information for features at various network layers by leveraging their gradient information. Additionally, we have developed a frequency feature selection strategy that judiciously extracts particular frequency information based on the gradient distribution of features. Lastly, to seamlessly merge the frequency-domain information at different frequencies with the spatial-domain information, we present a multi-frequency information fusion module, which significantly enhances the performance of pansharpening. Comprehensive evaluations performed on remote sensing image datasets highlight the exceptional performance of our proposed approach when contrasted with the most advanced pansharpening methods currently available.
The primary contributions of this paper are as follows:
We introduce a novel frequency information-adaptive filter module that intelligently extracts essential frequency-domain information for features at different network layers by leveraging their gradient information.We develop a judicious frequency feature selection strategy that selectively extracts specific frequency information based on the gradient distribution of features.We present a multi-frequency information fusion module to seamlessly integrate the frequency-domain information at different frequencies with the spatial-domain information.
Related work
In recent decades, researchers have extensively explored pansharpening, leading to the development of four primary methods [23]: Component Substitution (CS) [30], Multiresolution Analysis (MRA) [35], Variational Optimization (VO), and Deep Learning (DL). CS methods [15, 16, 19, 67, 68] focus on substituting the spatial component of LRMS images with the detailed spatial information from PAN images, aiming to preserve the original spectral content to the greatest extent within the transform domain. MRA techniques [8, 10, 11, 35] involve decomposing LRMS images into multi-resolution components and enhancing them with the high-frequency, detail-rich aspects of PAN images. VO approaches [53, 55–58] leverage prior knowledge to formulate constraints that guide the model, achieving pansharpening through an optimized algorithmic solution.
Pansharpening methodologies grounded in deep learning have captured widespread attention and marked significant advancements. Convolutional neural networks (CNNs), renowned for their exceptional proficiency in extracting both spectral and spatial information, have ascended to a pivotal role in pansharpening research [5, 6, 29, 31, 33, 34, 37, 39–42, 44–46]. Consequently, scholars have been delving into diverse strategies to augment the efficacy of pansharpening techniques within the spatial domain. However, despite the commendable performance of CNNs in this realm, they still grapple with challenges in capturing high-frequency details [22], often culminating in high-resolution multispectral images (HRMS) with inadequate texture details [22]. To mitigate this limitation, some researchers have begun to pivot towards frequency domain methods [13, 24, 38, 43, 49]. These approaches can more adeptly capture and preserve high-frequency information, thereby complementing the limitations inherent in spatial domain techniques. By extracting and amalgamating information within the frequency domain, researchers aspire to elevate the overall effectiveness of pansharpening, thus achieving superior image detail restoration throughout the pansharpening process.
CNN-based pansharpening methods
Masi et al [26]. pioneered this domain with the CNN-based PNN method, which employs a convolutional network to extract spatial features from PAN and LRMS images, subsequently integrating them into the upsampled LRMS image. However, the relatively simple architecture of PNN, consisting of merely three convolutional layers, results in slower convergence speeds. To address this, the DiCNN [50] method introduces skip connections between various layer feature maps, thereby mitigating gradient explosion and accelerating network convergence. Nevertheless, its simplistic network structure and limited number of parameters constrain its feature extraction capabilities. Zhang et al [44]. proposed the BDPN method, which enhances the accuracy of multi-scale detail information extraction in multispectral images. BDPN, inspired by traditional MRA methods, leverages a pyramid-structured bidirectional network to process both LRMS and PAN images. However, the extensive parameter size of BDPN complicates network training and limits the model’s generalization ability. MSDCNN [39] adopts a different strategy, utilizing two branches to extract information from both deep and shallow image layers. Each branch employs three convolution kernels of varying sizes to capture multi-scale features. By integrating features from different receptive fields, MSDCNN improves feature extraction accuracy. However, the versatility of its convolution kernels introduces increased uncertainty in learning deep and shallow features. FusionNet [51], another notable advancement, estimates a nonlinear injection model for detailed information using a deep network structure. It directly processes the original upsampled multispectral and pan images for differential detail extraction. The extracted details are then refined through multiple residual network blocks for further feature extraction and learning. The final output, when added to the upsampled multispectral image, yields the fusion image. This approach, while innovative, contends with the challenge of managing the complex interactions between different network layers.
Dual-domain pansharpening methods based on spatial and frequency domains
In light of advancements in CNN-based spatial domain pansharpening methods, researchers have explored incorporating frequency domain information to further enhance the quality of fused HRMS images. Man Zhou et al. [24] proposed a pansharpening method that fuses both domains, designing the Spatial-Frequency Information Integration Network (SFIIN), which extracts local spatial features from PAN and LRMS images while integrating frequency domain information to enhance their global context. This dual-domain fusion significantly boosts pansharpening performance. Subsequently, Yuan et al. [13] constructed the Pyramid-based Dual-Domain Network (PYDDN), injecting multi-scale frequency domain spatial details from the PAN image into multi-scale spatial information through a frequency feature pyramid. Hou et al. [38] proposed the BIM method, comprising two branches: band-aware local specificity modeling and Fourier global detail reconstruction. The local specificity modeling uses adaptive convolution kernels to process local discrepancies across spectral bands, while the global detail reconstruction branch leverages Fourier domain global modeling capabilities to recover lost global details. BIM exhibits exceptional local-to-global representation learning ability through the combination of spatial and frequency domains. Furthermore, Zhou et al. [43] introduced the Spatial-Frequency Information Integration Network (SFINet) and its improved version, SFINet++. The core component of SFINet, SFIB, fuses a spatial domain branch for processing local information, a frequency domain branch capturing global information using discrete Fourier transform, and a learning mechanism facilitating cross-domain information interaction. SFINet++ achieves significant performance improvements by incorporating lossless information fusion via reversible neural operators. Zhang et al. [49] proposed the Wavelet Domain Network (WINet), leveraging wavelet domain processing for efficient pansharpening. The Wavelet-Inspired Fusion Block (WFB) in WINet aims to achieve lossless information fusion through inter-subband interaction, while the High-frequency Enhancement Block (HEB) integrates subband information to enhance high-frequency features. By combining wavelet and CNN techniques, WINet effectively synthesizes high-quality HRMS images from spatial and frequency domain information. Lastly, Wang et al. [54] introduced an innovative pansharpening architecture where the Feature Extraction and Enhancement Module (FEM) utilizes fast Fourier convolution and attention mechanisms to form a hybrid of global and local receptive fields in the high-frequency domain. The Implicit Neural Alignment (INA) aligns multi-scale high-frequency features through accurate implicit neural representations, while the Pre-align Module develops an efficient trainable upsampling operator to address the inherent alignment challenge between PAN and MS images.
The significance of integrating frequency domain information with spatial domain features in dual-domain pansharpening methods is increasingly evident. This integration enables the production of high-quality HRMS images. By leveraging the complementary strengths of both spatial and frequency domains, these approaches demonstrate superior performance compared to traditional CNN-based spatial domain methods. However, existing methodologies often consider the CNN network in its entirety when integrating dual-domain information, overlooking the fact that different layers within a CNN network process frequency domain information of varying frequencies. This oversight potentially hampers the quality of dual-domain information fusion. Treating the CNN network as a monolith for frequency domain information fusion may not fully harness the capabilities of each layer, resulting in suboptimal dual-domain information fusion. To improve the effectiveness of dual-domain information fusion, we introduce FIAN, an innovative frequency information-adaptive network designed specifically for spatial-frequency domain pansharpening.
Materials and methods
Fig 2 presents the overall framework of FIAN. The process begins by upsampling a multispectral (MS) image to create a low-resolution multispectral (LRMS) image. This LRMS image, combined with a panchromatic (PAN) image, is fed into a convolutional neural network (CNN). In this framework, we propose a frequency information-adaptive filter module to extract frequency information from different features.
The overall framework of FIAN.The overall framework of the suggested frequency information-adaptive network for spatial-frequency domain pansharpening(FIAN). FIAFM stands for frequency information-adaptive filter module. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
The frequency information-adaptive filter module computes the discrete gradient of the CNN features, which, together with the PAN features, directs the filter extraction. These filters exhibit varying frequencies due to differences in the layer outputs. Subsequently, the frequency feature selection strategy identifies suitable features based on the gradient range. The frequency information-adaptive filter module extracts the frequency information from the selected features. The frequency information at various frequencies and their corresponding gradient calculations are then processed in the multi-frequency information fusion module. To mitigate the gradient vanishing problem commonly observed in convolutional neural networks, we have incorporated the Rectified Linear Unit (ReLU) activation function after every CNN layer in our network architecture.
Frequency information-adaptive filter module
The filter generation process in the frequency information-adaptive filter module comprises two crucial stages: the computation of discrete gradients from the features and the subsequent creation of frequency-discriminating filters corresponding to these gradients.
Discrete gradient.
To calculate the discrete gradients of CNN features, we apply a sequence of mathematical computations. Let be the feature tensor of the i-th selected layer of the CNN, where B, C, H, and W denote the batch size, number of channels, height, and width of the feature maps, respectively. The gradient computation process for is as follows:
First, we calculate the absolute differences between adjacent pixels in both horizontal and vertical directions to obtain the gradient tensors and :
Here, has dimensions [B, C, H, (W–1)] and has dimensions [B, C, (H–1), W].
To ensure the gradient tensors have the same dimensions as the original feature tensor, we apply zero-padding to and :
After padding, both and have dimensions [B, C, H, W]. Next, we compute the gradient magnitude using the Euclidean norm:
where has dimensions [B, C, H, W]. Finally, we perform global average pooling on the gradient magnitude tensor to obtain the mean of the scalar gradient :
The gradient computation process for the features of the i-th selected layer is summarized as:
where and represent the computation process in Eq 1, while Px and Py represent the padding process in Eq 2.
The computed value represents the gradient magnitude of the current CNN feature. A higher value of indicates a higher frequency of information present in the network feature. By examining the variations in values across different network layers, this approach quantitatively directs the extraction process of the frequency-discriminating filter.
Fig 2 depicts the discrete gradient calculation procedure utilized in the frequency information-adaptive filter components. The gradient computation result for the i-th module is denoted by . The aggregated gradient result from the previous (i – 1)-th module is represented by , while the aggregated result passed to the next + -th module is represented by . The mathematical formulation of is provided in Eq 6:
This expression represents the process of gathering discrete gradient information layer by layer during the network’s forward pass. is a vector that aggregates the gradient computation results from the first frequency information-adaptive filter module up to the (i – 1)-th module. is obtained by appending the gradient computation result of the current layer to the end of .
Frequency-discriminating filter.
This study presents an innovative frequency-discriminating filter that adaptively captures frequency information from the current feature based on its gradient .
In particular, the cutoff frequency of the frequency-discriminating filter is defined as k times the gradient , expressed as . Furthermore, we introduce an enhancement factor and a decay factor . These factors are employed to modify the upper and lower cutoff frequencies of the frequency-discriminating filter, respectively, as elaborated in Eq 7:
Here, and represent the enhanced upper cutoff frequency and the decayed lower cutoff frequency, respectively.
We develop a frequency-discriminating filter . Eq 8 presents the mathematical expression of the filter for each batch and channel:
Where D(u, v) represents the Euclidean distance from the point (u, v) to the center of the spectrum, defined by the formula . Aggregating across all batches and channels results in .
The two-dimensional Fast Fourier Transform (FFT) is first performed on the PAN image, as shown in Eq 9. Each PAN image can be viewed as a collection of B three-dimensional tensors with dimensions (C, H, W). For the two-dimensional image in the c-th channel of the b-th batch, a two-dimensional FFT is applied to obtain its spectrum .
Next, the amplitude spectrum and phase spectrum are extracted separately, as shown in Eq 10:
where and represent the modulus and argument of a complex number, respectively. The amplitude spectrum represents the intensity of each frequency component, while the phase spectrum contains the phase information of each frequency component.
After performing the Fourier transform on all batches and channels of the images, we obtain the representation of the PAN image in the frequency domain. The amplitude spectrum tensor and the phase spectrum tensor are composed of all and , respectively.
Finally, the is multiplied element-wise with the amplitude spectrum to obtain the filtered amplitude spectrum and the phase spectrum tensor , as shown in Eq 11:
The following is a concise summary of the frequency-discriminating filter process.
The coefficients k, , and are empirically determined to be 5.0, 1.5, and 0.8, respectively.
The frequency-discriminating filter introduced in this work adaptively modifies the cutoff frequencies according to the discrete gradients derived from the network layer features. This adaptive approach fully leverages the characteristics of the current network layer features, enabling the effective extraction of frequency domain information from PAN images within the appropriate range. Consequently, this enhances the accuracy and adaptability of high-frequency detail extraction.
Frequency feature selection strategy
As shown in Fig 1, the gradient information contained in the features of each CNN layer varies, reflecting information from the frequency domain at different frequencies. Existing pansharpening methods that combine spatial and frequency domains have not fully considered this characteristic. Instead, they treat the CNN network as a whole for frequency domain information fusion, which makes it difficult to fully exploit the unique advantages of each network layer, thereby affecting the performance of dual-domain information fusion.
To thoroughly analyze this problem, we randomly selected 10 samples from the training data and calculated the gradients of their ground truth (GT) images. The results indicate that the gradient range of the GT images is 45.5436-114.6202, as shown in Fig 3(a). Correspondingly, we performed the same gradient calculation on the features of a nine-layer CNN network. The gradient range of the nine-layer network features is 18.1816-82.3504, and as the network layers deepen, the feature gradients exhibit an upward trend. This suggests that when extracting image features, the CNN network prioritizes low-frequency information, which may lead to the loss of high-frequency details during training.
Schematic diagram of the frequency feature selection strategy.Schematic illustration of the frequency feature selection approach. (a) The spectrum of discrete gradients for ten randomly chosen GT data. (b) The procedure of identifying features that correspond to the numerical range of the GT discrete gradients and subsequently inputting these selected features into the frequency information-adaptive filter module for additional processing. FIAFM stands for frequency information-adaptive filter module. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
To mitigate the issue of high-frequency information loss, we earlier proposed a frequency information-adaptive filter module. This filter extracts information from different frequency ranges for different network layers and fuses it with spatial domain features. However, for certain CNN network layers (such as the first layer), the feature gradients are relatively low (only 18.1816), limiting the extracted frequency information. Moreover, this low-frequency information can already be well-extracted by the CNN network. Accordingly, this paper puts forward a frequency feature selection strategy founded on feature gradient intervals to enhance the network’s capacity for representation.
As depicted in Fig 3(b), the gradient values of the network layer features reveal that the frequency information contained in layers five through nine displays a higher degree of correspondence to the frequency information present in the GT. This indicates that the frequency information contained within the network layer features from the fifth to the ninth layers substantially affects the high-frequency details of the generated HRMS images. Consequently, we utilize this frequency information to refine and enhance the intricate high-frequency components of the HRMS images. This targeted feature selection strategy is designed to better supplement and enhance high-frequency details while preserving low-frequency information, ultimately improving the overall performance of dual-domain information fusion. We refer to this process as the range matching of network feature gradients, which involves aligning the numerical span of gradients derived from the CNN features with those obtained from the ground truth data.
Multi-frequency information fusion module
The multi-frequency information fusion module, depicted in Fig 4, integrates the amplitude spectra and phase spectra obtained from five distinct frequency-discriminating filters using a weighted combination, as formulated in Eq 13.
Multi-frequency information fusion module.The multi-frequency information fusion module amalgamates the results from a quintet of distinct frequency-discriminating filters with the feature output derived by the CNN network. In the schematic diagram, “OUT" represents the output features of the CNN network. FFT stands for Fast Fourier Transform, while IFFT represents the Inverse Fourier Transform.
The normalization of the discrete gradient yields the weight vector , as detailed in Eq 14:
The weighted fusion yields the amplitude spectrum and phase spectrum . Simultaneously, we perform a fast Fourier transform (FFT) on output features of the CNN network name OUT to obtain its amplitude spectrum and phase spectrum , a process previously described in Eq 9 and Eq 10.
Next, the fused amplitude spectrum , phase spectrum , and the OUT’s amplitude spectrum and phase spectrum are respectively fed into two 1 1 convolution, and the ReLU activation function is applied to the convolved features. Then, the two sets of activated amplitude spectrum features and phase spectrum features are concatenated separately, and finally, a 1 1 convolution is used to obtain the fused amplitude spectrum A and phase spectrum P.
Lastly, the fused amplitude spectrum A and phase spectrum P are input into the inverse Fourier transform (IFFT) to obtain the spatial-domain multi-frequency band fusion feature .
The multi-frequency information fusion module integrates data from various frequency ranges by performing a weighted combination of the amplitude and phase spectra, along with the spectral information of the LRMS image.
Result analysis and discussion
In this section, we shall evaluate the proposed method and juxtapose its results with those of recent, competitive approaches. The data employed in these evaluations originate from datasets procured from the WorldView-3 (WV3), WorldView-2 (WV2), and QuickBird (QB) satellites. Initially, we will delineate the specifics of the datasets utilized for this analysis in greater detail. Following this, we will outline the evaluation metrics and the particulars of the training process. Subsequently, we will exhibit the test results of the proposed method alongside those of the comparative methods across the three datasets, incorporating both quantitative metrics and visual illustrations under low-resolution and full-resolution conditions, to affirm the effectiveness of the proposed FIAN. Ultimately, the efficacy of the suggested FIAN and its component modules will be meticulously examined via an extensive series of ablation studies.
Dataset
To evaluate our pansharpening method, we used an extensive dataset, including 4-band datasets from QB and 8-band datasets from WV3 and WV2. The 4-band sets include standard colors: red, green, blue, and near-infrared; the 8-band sets add coastal, yellow, red-edge, and an additional near-infrared band. The spatial resolution ratio between PAN (panchromatic) and MS (multispectral) images is 4:1. Owing to the absence of ground truth images, we spatially degraded the datasets using the Wald’s protocol [47], which involves downsampling the original images by a factor of 4 through filters aligned with each satellite’s modulation transfer function (MTF). In our network, the simulated datasets’ panchromatic and multispectral images serve as inputs, while the original multispectral images act as reference data for training. The datasets used in this study are from the PanCollection dataset [17], which contains detailed data descriptions. For example, in the WV3 dataset, we used 9,714 pairs of PAN-LRMS-GT data for network training, where the PAN images have a size of 64 64 1, the LRMS images have a size of 16 16 8, and the GT images have a size of 64 64 8. For low-resolution evaluation, we selected 20 pairs of PAN-LRMS-GT data for testing, where the PAN images have a size of 256 256 1, the LRMS images have a size of 64 64 8, and the GT images have a size of 256 256 8. For full-resolution evaluation, we selected 20 pairs of PAN-LRMS images for testing, where the PAN images have a size of 512 512 1 and the LRMS images have a size of 128 128 8.
Training details and parameters
In this study, we developed our network using the PyTorch 2.0 framework on a Linux system, with a single NVIDIA GeForce GTX 3090 GPU supporting our hardware needs. Our software setup also included Python 3.10. Throughout the training phase, we conducted 1000 epochs. The learning rate started at 3 10^−4^ for the initial 500 epochs, then reduced to one-tenth of that rate for the latter half. We utilized the ADAM optimizer [52] to manage parameter optimization, maintaining a batch size of 32 and setting at 0.9 and at 0.999.
Comparison methods and quantitative metrics
To assess the performance of our proposed method, we performed both qualitative and quantitative comparisons against current state-of-the-art pansharpening techniques. We compared our method with two main types of approaches: CNN-based spatial domain methods (including PNN [26], MSDCNN [39], DRPNN [31], DiCNN [50], BDPN [44], FusionNet [51] and dual-domain methods that integrate spatial and frequency domains (including WINet [49], PYDDN [13], SFINet [43], BIM [38]). All competing methods were trained on identical training sets, using parameters specified in their respective original publications. To ensure fairness and consistency in our evaluations, all methods were tested under the same hardware and software conditions.
This paper provides a comprehensive assessment of the performance of the methods mentioned earlier, using both reduced-resolution and full-resolution datasets. For the reduced-resolution tests, we employ several metrics: ERGAS [21], SAM [4], SCC [32], and Q2n [36]. These metrics help quantify the quality of the results. For the full-resolution tests, we use the , DS, and QNR [1] to evaluate the performance of all methods.
Performance comparison with WV3 data
In this section, we conduct a comprehensive evaluation and comparison of the proposed method and benchmark methods on the WV3 dataset, considering both reduced-resolution and full-resolution aspects.
First, we assess the differences between the predicted pansharpening images and the ground truth (GT) images through reduced-resolution evaluation. Table 1 presents the objective assessment metric results for all compared methods and our proposed model across 20 WV3 test samples. The table demonstrates that our model achieves the best performance across all metrics, showcasing the effectiveness and superiority of our approach. Compared to benchmark pansharpening methods, our model better preserves the spectral information of multispectral images while effectively enhancing spatial resolution, resulting in high-quality image fusion.
Table 1: Quantitative metrics of reduced-resolution WV3 data.
Fig 5 provides a visual comparison of the pansharpening results from all compared methods to further illustrate the performance differences. We selected a representative area of the image (marked with a green box), enlarged it, and positioned it in the upper left corner of the original image for detailed observation. Close examination of the enlarged area reveals that the fusion results produced by our model closely resemble the ground truth image, exhibiting excellent performance in texture, edges, and color reproduction.
Visual comparisons of reduced-resolution WV3 data.The visual comparisons of fusion results from various methods applied to reduced-resolution data from the WorldView-3 satellite. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
Secondly, we assess the pansharpening performance of various methods on actual data through full-resolution evaluation. Table 2 displays the objective assessment metric results for all compared methods and our proposed model across 20 full-resolution WV3 test samples. The results in the table demonstrate that our model achieves the best performance across all metrics, highlighting the effectiveness and superiority of our approach.
Table 2: Quantitative metrics of full-resolution WV3 data.
Fig 6 provides a visual comparison of the pansharpening results from all methods evaluated, further illustrating performance differences. We selected a representative area of the image (marked with a green box), enlarged it, and placed it in the lower left corner of the original image for detailed observation. Upon close examination of the enlarged area, it is evident that the fusion results produced by our model excel in texture, edge definition, and color reproduction.
Visual comparisons of full-resolution WV3 data.The visual comparisons of fusion results from various methods applied to full-resolution data from the WorldView-3 satellite. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
Performance comparison with WV2 data
In this section, we conduct a comprehensive evaluation and comparison of the proposed method and benchmark methods on the WV2 dataset, considering both reduced-resolution and full-resolution aspects.
First, we assess the differences between the predicted pansharpening images and the ground truth (GT) images through reduced-resolution evaluation. Table 3 presents the objective assessment metric results for all compared methods and our proposed model across 20 WV2 reduce-resolution test samples. The table demonstrates that our model achieves the best performance across all metrics, showcasing the effectiveness and superiority of our approach. Compared to benchmark pansharpening methods, our model better preserves the spectral information of multispectral images while effectively enhancing spatial resolution, resulting in high-quality image fusion.
Table 3: Quantitative metrics of reduced-resolution WV2 data.
Fig 7 provides a visual comparison of the pansharpening results from all compared methods to further illustrate the performance differences. We selected a representative area of the image (marked with a green box), enlarged it, and positioned it in the lower left corner of the original image for detailed observation. Close examination of the enlarged area reveals that the fusion results produced by our model closely resemble the ground truth image, exhibiting excellent performance in texture, edges, and color reproduction.
Visual comparisons of reduced-resolution WV2 data.The visual comparisons of fusion results from various methods applied to reduced-resolution data from the WV2 satellite. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
Secondly, we assess the pansharpening performance of various methods on actual data through full-resolution evaluation. Table 4 displays the objective assessment metric results for all compared methods and our proposed model across 20 full-resolution WV2 test samples. The results in the table demonstrate that our model achieves the best performance across all metrics, highlighting the effectiveness and superiority of our approach.
Table 4: Quantitative metrics of full-resolution WV2 data.
Fig 8 provides a visual comparison of the pansharpening results from all methods evaluated, further illustrating performance differences. We selected a representative area of the image (marked with a green box), enlarged it, and placed it in the lower left corner of the original image for detailed observation. Upon close examination of the enlarged area, it is evident that the fusion results produced by our model excel in texture, edge definition, and color reproduction.
Visual comparisons of full-resolution WV2 data.The visual comparisons of fusion results from various methods applied to full-resolution data from the WV2 satellite. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
Performance comparison with QB data
In this section, we conduct a comprehensive evaluation and comparison of the proposed method and benchmark methods on the QB dataset, considering both reduced-resolution and full-resolution aspects.
First, we assess the differences between the predicted pansharpening images and the ground truth (GT) images through reduced-resolution evaluation. Table 5 presents the objective assessment metric results for all compared methods and our proposed model across 20 WV2 reduce-resolution test samples. The table demonstrates that our model achieves the best performance across all metrics, showcasing the effectiveness and superiority of our approach. Compared to benchmark pansharpening methods, our model better preserves the spectral information of multispectral images while effectively enhancing spatial resolution, resulting in high-quality image fusion.
Table 5: Quantitative metrics of reduced-resolution QB data.
Fig 9 provides a visual comparison of the pansharpening results from all compared methods to further illustrate the performance differences. We selected a representative area of the image (marked with a green box), enlarged it, and positioned it in the lower left corner of the original image for detailed observation. Close examination of the enlarged area reveals that the fusion results produced by our model closely resemble the ground truth image, exhibiting excellent performance in texture, edges, and color reproduction.
Visual comparisons of reduced-resolution QB data.The visual comparisons of fusion results from various methods applied to reduced-resolution data from the QB satellite. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
Secondly, we assess the pansharpening performance of various methods on actual data through full-resolution evaluation. Table 6 displays the objective assessment metric results for all compared methods and our proposed model across 20 full-resolution QB test samples. The results in the table demonstrate that our model achieves the best performance across all metrics, highlighting the effectiveness and superiority of our approach.
Table 6: Quantitative metrics of full-resolution QB data.
Fig 10 provides a visual comparison of the pansharpening results from all methods evaluated, further illustrating performance differences. We selected a representative area of the image (marked with a green box), enlarged it, and placed it in the lower left corner of the original image for detailed observation. Upon close examination of the enlarged area, it is evident that the fusion results produced by our model excel in texture, edge definition, and color reproduction.
Visual comparisons of full-resolution QuickBird data.The visual comparisons of fusion results from various methods applied to full-resolution data from the QB satellite. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
Ablation study
To further substantiate the efficacy of FIAN, we performed ablation experiments. This subsection delves into the importance of the frequency feature selection strategy, the multi-frequency information fusion module, and the frequency information-adaptive filter module.
Table 7 presents seven ablation experimental comparison strategies for the aforementioned modules. First, an ablation experiment was conducted on the frequency feature selection strategy. In FIAN, the last five network layers were selected as input for the frequency information-adaptive filter module by matching the gradient range of the ground truth (GT) with that of the nine-layer CNN network features. The ablation of the frequency feature selection strategy involved testing the impact of the first five and the last five CNN network layers on pansharpening performance, presented in Table 8 as (I), (II), (III), and (IV), with (IV) representing our proposed FIAN method.
Table 7: The ablation protocols for the suggested FIAN approach, in which the frequency information-adaptive filter module is denoted by FIAFM and the multi-frequency information fusion module is symbolized by MFIFM.
Table 8: Quantitative metrics of ablation methods on reduced-resolution WV3 data.
Following this, ablation experiments were conducted on the frequency information-adaptive filter module (FIAFM) and the multi-frequency information fusion module (MFIFM), precisely (V), (VI), and (VII). Notably, (VII) employs the same strategy as (I).
We conducted a series of ablation experiments on reduced-resolution data from the WorldView-3 satellite, with the objective metrics presented in Table 8. By comparing (I), (II), and (III) with (IV), we validated the effectiveness of the proposed frequency feature selection strategy; notably, (IV) achieved the best results across all metrics. The comparison between (V), (VI), and (VII) with (IV) further confirmed the efficacy of the frequency information-adaptive filter module (FIAFM) and the multi-frequency information fusion module (MFIFM).
Fig 11 provides a visual comparison of the pansharpening results from all methods. Our proposed FIAN method (IV) most closely resembles the ground truth (GT) images, displaying superior spatial texture, edge clarity, and color reproduction compared to other scenarios.
Visual comparisons of ablation methods on reduced-resolution WV3 data.Visual comparisons of the effects from all ablation methods applied to reduced-resolution data from the WorldView-3 satellite. Reprinted from https://github.com/liangjiandeng/PanCollection under a CC BY license, with permission from Liangjian Deng, original copyright 2025.
Conclusions
This study introduces the frequency information-adaptive network (FIAN), a groundbreaking pansharpening technique that seamlessly combines information from both the spatial and frequency domains. FIAN incorporates a frequency information-adaptive filter module that adaptively creates filters with varied frequency attributes by leveraging the gradients of distinct CNN features. FIAN also incorporates a frequency feature selection approach that aims to optimally complement and enhance high-frequency details while retaining low-frequency information. This targeted strategy ultimately elevates the overall performance of dual-domain information fusion by striking a balance between preserving crucial low-frequency components and enriching high-frequency nuances. Moreover, the multi-frequency information fusion module amalgamates data from diverse frequency ranges by executing a weighted combination of the amplitude and phase spectra, in conjunction with the spectral information derived from the LRMS image. This integrated approach ensures a comprehensive assimilation of information spanning the entire frequency spectrum, thereby enhancing the richness and fidelity of the fused output. Comprehensive evaluations performed on both downscaled and original resolution datasets showcase FIAN’s superior performance compared to cutting-edge pansharpening techniques. The proposed method shows better performance on quantitative metrics and produces visually appealing pansharpening images that preserve spatial details and spectral fidelity. While the proposed FIAN method demonstrates significant advantages in remote sensing pansharpening, the underlying adaptive frequency selection strategy can also be extended to other imaging domains, such as digital photography and image super-resolution.
Despite the demonstrated effectiveness of our method, certain limitations remain. First, the proposed neural network architecture is relatively complex, leading to longer training times and higher computational demands. Second, while the adaptive band-pass filtering strategy effectively suppresses noise in general, it currently employs a relatively simple model for frequency band selection. Future research could include more targeted modeling of noise characteristics to further enhance robustness. These limitations represent important directions for future improvements.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Vivone G, Alparone L, Chanussot J, Dalla Mura M, Garzelli A, Licciardi G, et al. A critical comparison among pansharpening algorithms. IEEE Trans Geosci Remote Sens. 2014;53:2565–86.
- 2Li X, Yan H, Xie W, Kang L, Tian Y. An improved pulse-coupled neural network model for pansharpening. Sensors (Basel). 2020;20(10):2764. doi: 10.3390/s 20102764 32408666 PMC 7294424 · doi ↗ · pubmed ↗
- 3Liu H, Deng L, Dou Y, Zhong X, Qian Y. Pansharpening model of transferable remote sensing images based on feature fusion and attention modules. Sensors (Basel). 2023;23(6):3275. doi: 10.3390/s 23063275 36991987 PMC 10055757 · doi ↗ · pubmed ↗
- 4Yuhas R, Goetz A, Boardman J. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In: JPL Summaries of the Third Annual JPL Airborne Geoscience Workshop. 1992. p. 147–9.
- 5Jin X, Feng Y, Jiang Q, Miao S, Chu X, Zheng H, et al. Upgan: an unsupervised generative adversarial network based on u-shaped structure for pansharpening. ISPRS Int J Geo-Inf. 2024;13(222).
- 6Huang W, Feng J, Wang H, Sun L. A new architecture of densely connected convolutional networks for pan-sharpening. ISPRS Int J Geo-Inf. 2020;9(242).
- 7Li H, Jing L, Tang Y. Assessment of pansharpening methods applied to World View-2 imagery fusion. Sensors (Basel). 2017;17(1):89. doi: 10.3390/s 17010089 28067770 PMC 5298662 · doi ↗ · pubmed ↗
- 8Aiazzi B, Alparone L, Baronti S, Garzelli A, Selva M. Mtf-tailored multiscale fusion of high-resolution ms and pan imagery. Photogramm Eng Remote Sens. 2006;72:591–6.