Saliency-enhanced infrared and visible image fusion via sub-window variance filter and weighted least squares optimization

TL;DR

This paper introduces a new image fusion method that improves the clarity and visibility of important features in infrared and visible images.

Contribution

The novel method uses sub-window variance filter and weighted least squares optimization for better image fusion without deep learning.

Findings

The method outperforms nine state-of-the-art techniques in image quality and salient feature highlighting.

It preserves texture details and avoids image distortion without needing large training datasets.

Future work will address handling complex scenes through adaptive optimization and deep learning integration.

Abstract

This paper proposes a novel method for infrared and visible image fusion (IVIF) to address the limitations of existing techniques in enhancing salient features and improving visual clarity. The method employs a sub-window variance filter (SVF) based decomposition technique to separate salient features and texture details into distinct band layers. A saliency map measurement scheme based on weighted least squares optimization (WLSO) is then designed to compute weight maps, enhancing the visibility of important features. Finally, pixel-level summation is used for feature map reconstruction, producing high-quality fused images. Experiments on three public datasets demonstrate that our method outperforms nine state-of-the-art fusion techniques in both qualitative and quantitative evaluations, particularly in salient target highlighting and texture detail preservation. Unlike deep learning-based approaches, our method does not require large-scale training datasets, reducing dependence on ground truth and avoiding fused image distortion. Limitations include potential challenges in handling highly complex scenes, which will be addressed in future work by exploring adaptive parameter optimization and integration with deep learning frameworks.

Figures

Visualization of fusion results for partial methods.

The overall pipeline of the method proposed in this paper.

Visualization results of 10 methods on “ Marne_04” image pairs in the TNO dataset.

Visualization results of 10 methods on “Kaptein_1123” image pairs in the TNO dataset.

Visualization results of 10 methods in the RoadScene dataset.

Qualitative comparison of the 10 methods on images from the LLVIP dataset.

Qualitative comparison results of ablation experiments on the TNO dataset, where (a) w/o SVF denotes the use of FPDE as the decomposition method and (b) w/o SW denotes the use of average method as the fusion strategy.

Equations

Full text

1. Introduction

High-quality image data is crucial for advancing detection technology and artificial intelligence in computer vision. However, limited by differences in imaging mechanisms, a single sensor often fails to capture the complete information of a scene [1]. For example, the imaging principle of infrared sensors is thermal radiation imaging, and the captured images tend to highlight thermal targets such as humans and vehicles, which are less affected by changes in light. Nevertheless, the drawbacks are more noise in the image, blurring of texture details and poor visualization. In contrast, the imaging principle of visible sensors is reflected light imaging, which captures images with a textured appearance and rich background information. However, it tends to be susceptible to light variations and suffers from severe information loss. It is necessary to effectively integrate images containing different modal feature information to generate a fused image with clear vision, prominent features and rich information. Therefore, infrared and visible image fusion (IVIF) techniques have emerged [2–6]. These techniques aim to enhance perceptual image quality by combining the strengths of both modalities, which is closely related to the field of perceptual image quality assessment [7–9]. In addition, tasks such as multi-exposure image fusion [10,11], multi-focus image fusion [12,13], multi-modal medical image fusion [14,15], and multi-spectral image fusion [16,17] have been derived based on the differences in information present in the images captured by the sensors. Moreover, High-quality images generated by image fusion techniques are widely employed in various fields, including object detection [18], target tracking [19], and remote sensing detection [20,21]. More importantly, recent advancements in multimodal fusion and visual saliency have demonstrated the importance of integrating complementary information and perceptual quality enhancement [22–24]. These studies provide valuable insights into the role of saliency-driven approaches and multimodal data integration, which inspire our work to improve the quality and interpretability of infrared and visible image fusion.

With the rapid development and iterative updating of filter theory and deep learning techniques, the exploration on image fusion techniques has become a research hotspot. At present, image fusion algorithms can generally be classified into two broad categories according to the type of algorithm, i.e., traditional image fusion methods based on mathematical theory and fusion methods based on deep learning. Among them, the traditional image fusion algorithms primarily include methods based on multi-scale transform (MST) [3,25], subspace-based [26] and sparse representation (SR) -based [27]. Such methods usually do not require heavy and complex training and often rely on manually designed fusion strategies. For example, MST-based methods often perform image decomposition through some kind of filter, such as weighted least squares filter [28], cross-bilateral filter [29], etc., and then fusion and reconstruction of the multiscale layers are performed by hand-designed fusion strategies. Subspace-based techniques frequently map characteristics of high-dimensional data into lower-dimensional spaces by some kind of dimensionality reduction method for further feature fusion [30]. SR-based methods extract image features and improve the performance of image fusion by learning an overcomplete dictionary [31].

Moreover, owing to the robust capabilities for feature extraction and representation inherent in deep neural networks, more refined and effective image fusion results are realized. Based on the specific architecture and operational principles of the network, deep learning based networks include three main categories, namely, convolutional neural network (CNN) [32], auto-encoder (AE) [33] and generative adversarial network (GAN) [34] based methods. The CNN-based method are used to generate impressive fusion results through specially designed feature extraction networks and reconstruction networks [35]. The AE learns an efficient representation of the source image through an encoder and reconstructs it through a decoder to generate a complete fused image [36]. In the GAN-based method, the generator is tasked with creating a fused image, and the discriminator ensures that this image encapsulates richer feature information from the source by comparing it directly [37].

Although most existing algorithms produce relatively favorable fusion results, certain limitations remain to be addressed. 1) many traditional methods, such as those based on MST [25] or SR [27], rely on handcrafted fusion rules that may not effectively enhance salient features while preserving texture details. 2) Existing fusion methods obtain satisfactory fused images by designing complex fusion networks, however, the enhancement of salient features is neglected while ensuring the richness of feature information. As shown in Fig 1, both SDNet [32] and ICAFusion [38] result in a fused image that contains complete information, but falls short in highlighting the salient features of thermal radiation, as shown in the purple box. 3) Deep learning-driven fusion methods often require extensive datasets of high-quality natural images to train networks, aiming to make the fusion model have excellent fusion performance and robustness. These limitations highlight the need for a new approach that can effectively enhance salient features without relying on large-scale training data.

Visualization of fusion results for partial methods.

To tackle the aforementioned challenges, we presents a new image fusion method based on sub-window variance filter (SVF) and weighted least squares optimization (WLSO). First, in order to more comprehensively separate the salient features, texture and background information into different band layers, a decomposition scheme based on a SVF is proposed to separate the attribute features of different scales into the detail and base layers, respectively. Next, a saliency map measure scheme based on WLSO is designed to compute the weight map of important information in the source image. Then, the obtained weight maps are weighted and summed at the detail and base layers, respectively, to further enhance and highlight the displayability of different features. Finally, feature reconstruction is performed by pixel-level summation to obtain the desired fusion result.

Specifically, the primary objective of this study is to address the limitations of existing fusion methods, particularly the inadequate enhancement of salient features and the reliance on large-scale training datasets. Our proposed method aims to effectively separate and highlight salient features while preserving texture details, without requiring extensive training data, thereby improving the quality and interpretability of infrared and visible image fusion.

The primary contributions of this paper are summarized as follows:

(1) A decomposition method based on SVF is proposed. The significant and structural features in the input image are separated by calculating and comparing the global variance and local variance, so as to effectively decompose different modal features into detail and base layers.(2) A visual saliency map measure scheme based on WLSO is proposed for different modal features. By calculating the feature intensity distribution of the global region in the source image and then optimizing it using the weighted least squares method, the pixel intensity weight map is obtained.(3) We conducted extensive comparative experiments with nine advanced fusion methods on three publicly available datasets. The experimental results confirm that the proposed method excels in highlighting significant thermal radiation features while preserving texture details.

In short, the innovation of our study lies in the combination of the SVF for decomposition and the WLSO for saliency map measurement. This combination allows our method to effectively separate and enhance salient features while preserving texture details, addressing the limitations of existing methods that either neglect salient feature enhancement or rely on large-scale training datasets.

The subsequent sections of this paper are structured as follows. Section 2 presents an overview of conventional and deep learning-driven image fusion methodologies, followed by a detailed description of our novel method in Section 3. Comparison experiments and ablation experiments on three datasets are detailed in Section 4 to validate the effectiveness of the proposed method. The conclusion can be found in Section 5. Limitations and future work in Section 6.

2. Related work

2.1 Conventional image fusion methods

Currently, the mainstream conventional image fusion methods includes MST [3,39], subspace [27,40] and SR [41] -based methods.

In the fusion method based on MST, Tang et al. [3] employed a weighted least squares filter to separate the input image into the base and detail layers, and designed a saliency map measurement based on an adaptive weight assignment strategy and a SVF to enhance the fused base and detail layers, respectively, and ultimately output the fused results with significant gradient features. Liu et al. [42] have extracted infrared target features and texture details from the original images by performing coarse-scale decomposition on infrared images and fine-scale decomposition on visible images, respectively. In addition, Zhang et al. [39] proposed a two-scale decomposition framework based on an average filter to effectively preserve the gradient variations and the thermal radiation features. Li et al. [43] proposed a fusion method (IVFusion) based on MST and paradigm optimization, which takes the pre-fused image as a reference and introduces structural similarity to assess the validity of the detail information, combined with L2 norm optimization method in order to generate the fused image.

In subspace-based fusion methods, typically, the proposed FPDE [26] utilizes principal component analysis (PCA) to fuse high-frequency detail information, which provides ideas for subsequent research. Subsequently, Liu et al. [44] used an improved tensor robust PCA to downscale the image to different subspace regions to achieve effective separation of different modal features in the source image. In addition, Omer et al. [40] captured image salient features via independent component analysis (ICA) and combined it with Chebyshev polynomial approximation to remove the noise, which effectively suppresses the introduction of extraneous details, ensuring that the fused result is free from redundant information from the source image. In their study [45], Mou et al. applied nonnegative matrix factorization (NMF) for feature extraction, effectively enhancing the salient features while retaining the essential texture details.

In the SR-based method, building upon convolutional SR, Wohlberg et al. [41] introduced an image fusion method that offers greater flexibility and efficiency in fusion processes. Aishwarya et al. [27] added supervised dictionary learning to the SR model, which effectively reduced the number of dictionaries trained. To obtain effective information from the source images, Li et al. [46] proposed a fusion method based on latent low-rank representation. However, this method faces significant computational efficiency issues due to the extensive mathematical calculations required to obtain the optimal solution. In addition, Li et al. [47] combined online robust dictionary learning with a guided filter, and used the idea of patch-based clustering to classify similar feature pixels, which achieved to retain the critical information in the input image.

2.2. Deep learning fusion methods

Over the past decade, deep neural networks have been increasingly utilized by researchers for their robust feature extraction and representation capabilities [48–51]. Nowadays, research on image fusion algorithms based on deep learning architectures is also gradually becoming a new research hotspot. Depending on the differences in network architecture, fusion methods based on deep learning are categorized into three main classes, i.e., fusion methods based on CNN [32,35], AE [33], and GAN [52].

In the CNN-based method, Hou et al. [35] designed an unsupervised end-to-end IVIF network with a structure consisting of cascaded dense blocks, and achieved feature reconstruction with stacked convolution operations to generate fusion results. The method effectively and adaptively fuses thermal radiation features and texture details with simultaneous suppression of noise interference. Zhao et al. [53] proposed a algorithm for unfolded image fusion (AUIF), which is a model that contains two encoders and a decoder, and efficiently generates a fused image that not only captures salient targets but also preserves clear and discernible detail. Furthermore, aiming to accomplish an image fusion task that is not only more efficient but also highly accurate, Liu et al. [54] proposed a convolution-based lightweight pixel-level unified fusion network. PMGI utilizes multi-gradient information to guide the image fusion process and employs CNN to extract and fuse multi-scale features [55].

In the AE-based method, Li et al. [36] designed dense blocks as encoder structures for the first time, which effectively improved the information flow in the network and prevented information loss. Additionally, Li et al. [56] have pioneered the incorporation of an intricate nested connection architecture within fusion networks, called NestFuse. The nested architecture adequately preserves and utilizes features at different scales, and a fusion strategy based on spatial attention and channel attention models is designed for fusing multi-scale depth features. Building upon NestFuse, Li et al. [57] proposed a learnable residual fusion network to replace manually designed fusion strategies, effectively addressing the issue of insufficient information utilization. Xing et al. [58] introduced a compressed fusion network named CFNet, which incorporates the idea of image compression based on variational autoencoders to achieve joint optimization of image fusion and data compression.

In the GAN-based approach, Ma et al. [37] proposed a fusion network with dual discriminators, which effectively solved the problem of the lack of infrared image information. Moreover, Li et al. [52] proposed a fusion network that combines the attention mechanism with a GAN, called AttentionFGAN. In this method, the information within the input image is retained to the maximum extent, with key details being effectively brought into focus. Le et al. [59] first proposed a fusion network of continuously learned GAN, named UIFGAN, for a unified image fusion task. To tackle the problem of global information loss, Wu et al. [60] proposed a globally aware generative adversarial fusion network and introduced differential image loss to further constrain the generator to learn important information from the source images.

3. Methodology.

We initially offer a comprehensive overview of the proposed methodology. Then, we describe in detail the proposed decomposition scheme and the concrete realization process of the fusion strategy.

3.1. Overall overview

As illustrated in Fig 2, the overall framework of the proposed method is systematically displayed. The specific flow of the proposed method is as follows:

The overall pipeline of the method proposed in this paper.

(1) Image Decomposition

Due to the richness and diversity of the information contained in the source images. Therefore, we propose an image decomposition scheme based on SVF [61] aiming at separating different feature information into different band layers. The base layer contains the overall contrast and background features, while the detail layer contains texture and edge information. The decomposition process is given by:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ are the base layer, $[eqn]$ and $[eqn]$ are the detail layer, SVF(∙) denotes the SVF operator, and $[eqn]$ and $[eqn]$ denote the infrared image and the visible image, respectively. To ensure clarity, the SVF operator is explained in detail in Section 3.2, including the mathematical formulation and implementation steps.

(2) Visual Saliency Map and Weight Calculation

In order to make the key information in the source image can be fully preserved and fully presented in the fusion result. We designed a visual saliency map measure scheme based on WLSO. The whole process is represented by the following steps:

Step 1: Compute the visual saliency maps for the infrared and visible images using the Visual Saliency Maping (VSM) operator:

$$[eqn]$$

Step 2: Optimize the saliency maps using Weighted Least Squares Optimization (WLSO) to obtain the weight maps:

$$[eqn]$$

Step 3: Calculate the final weight coefficients:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ denote the saliency maps of the source images, respectively. $[eqn]$ denotes the operator for computing the visual saliency maps. $[eqn]$ and $[eqn]$ denote the weight maps after weighted least squares optimization, respectively. This ensures that the fusion process adaptively balances the contributions of the infrared and visible images based on their saliency information. $[eqn]$ denotes the weighted least squares optimization operator. $[eqn]$ denotes the final weight coefficients. The computation of the saliency maps and the optimization process are further elaborated in Section 3.3.

(3) Fusion of Base and Detail Layers

The obtained weight maps are weighted and summed with the base and detail layers, respectively, with the purpose of fully integrating the private modal information in the base and detail layers. This process is denoted as:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ and $[eqn]$ denote the fused base and detail layer, respectively.

(4) Final Fusion Result

The fused base and detail layers are summed at the element level to obtain the desired result $[eqn]$ , denoted as:

$$[eqn]$$

3.2. Decomposition method

The sub-window variance filter (SVF) is an edge-preserving filter constructed using local edge statistics information, which improves the edge-awareness of the filter by computing the information of the spatial statistics of the image [61]. The SVF-based decomposition scheme separates the source image into base and detail layers. Specifically, the result of the SVF can be viewed as a linear combination of the input image and its smoothed filtered result. Here, it is assumed that the image patch of the input image centered at pixel $[eqn]$ is $[eqn]$ , and the result after the sub-window variance filter is $[eqn]$ , which can be expressed as:

$$[eqn]$$

where the weight coefficient $[eqn]$ is used to control the contribution of the input image block $[eqn]$ in the filtered image $[eqn]$ , and $[eqn]$ . $[eqn]$ is a generalized box filter.

After Eq (7), it is able to limit the pixel value of the filtered image $[eqn]$ between the pixel value and the mean value of the original image, thus effectively eliminating the occurrence of oversharpening in the bilateral filter [28]. Since F(∙) is a box filter, Eq. (7) can be further rewritten as:

$$[eqn]$$

where $[eqn]$ denotes the receptive domain of the filter when centered at pixel $[eqn]$ , and $[eqn]$ denotes the intensity value of the $[eqn]$ th pixel in the receptive domain.

The computation of weighting factor $[eqn]$ is explicitly described as follows: Divide the filtering region $[eqn]$ into four equal-sized sub-windows $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ . This division is based on the need to capture local variance information effectively in different spatial directions (e.g., horizontal, vertical, and diagonal). The choice of four sub-windows provides a balanced representation of local edge statistics, ensuring that the filter can adaptively preserve edge features while smoothing homogeneous regions. Calculate the set of local variance values corresponding to the four subregions $[eqn]$ , respectively. As well as the global variance value $[eqn]$ of the region to be filtered $[eqn]$ . Then the weighting factor $[eqn]$ can be defined as:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ . $[eqn]$ is a regularization parameter. Here, when $[eqn]$ , then $[eqn]$ , and edge features will be mostly preserved. When $[eqn]$ , then $[eqn]$ and the edge is fully preserved. Moreover, when $[eqn]$ is a minimal value such that $[eqn]$ , most of the region in the image is smoothed.

Based on the above description, from Eqs. (8) to (10), the operator $[eqn]$ for the sub-window variance filter can be expressed as:

$$[eqn]$$

where $[eqn]$ denotes the base layer which contains most of the background features and overall contrast information. Texture details and edge features, which constitute the high-frequency information in an image, can be precisely extracted by calculating the differences between the original and the corresponding base layer. In other words, The detail layer D is extracted by subtracting the base layer from the original image, denoted as:

$$[eqn]$$

where $[eqn]$ contains high-frequency information such as texture and edges.

3.3 Fusion strategy

First, the visual saliency map measure scheme is a classical saliency map computation that aims to establish the contrast intensity present between different pixels in an image to define the saliency map [62]. Assuming that the intensity value of the $[eqn]$ th pixel in the input image $[eqn]$ is denoted as $[eqn]$ , the saliency map $[eqn]$ of pixel $[eqn]$ is denoted as:

$$[eqn]$$

where $[eqn]$ stands for the total number of pixels and $[eqn]$ indicates the absolute operator. Eq. (13) can be further rewritten as:

$$[eqn]$$

Then, $[eqn]$ is normalized to $[eqn]$ ensures that the saliency values are comparable across different images. Eq. (14) that represents the $[eqn]$ operator.

To further clarify the fusion strategy, the optimization process using the weighted least squares method is described in detail.

Specifically, in order to further enable the features on the saliency map to be mapped to the fusion results and to exclude potential redundant information and noise interference, for this purpose, we have optimized the saliency map using a weighted least squares method [28]. We take the salient map $[eqn]$ as input, and the optimization function can be expressed as,

$$[eqn]$$

The optimization function in Eq. (15) aims to minimize the difference between the saliency map $[eqn]$ and the weight map $[eqn]$ , while penalizing large gradients in $[eqn]$ . This ensures that the weight map is smooth and free from noise. In Eq. (15), where $[eqn]$ denotes the balance coefficient, which controls the trade-off between fidelity to the saliency map and smoothness of the weight map. The choice of λ is based on ablation experiments to achieve optimal performance, $[eqn]$ denotes the spatial location of the pixel, and $[eqn]$ and $[eqn]$ denote the weight matrices of the gradient in the $[eqn]$ and $[eqn]$ directions, respectively. Next, Eq. (15) can be converted to matrix form as follows.

$$[eqn]$$

where $[eqn]$ and $[eqn]$ are diagonal weight matrices and $[eqn]$ and $[eqn]$ are matrix representations of discrete difference operators. Let the derivative of $[eqn]$ be zero, i.e.,

$$[eqn]$$

According to Eq. (17), the solution representation of the optimization goal can be obtained as:

$$[eqn]$$

where $[eqn]$ is the unit matrix and $[eqn]$ . From Eq. (18), the operator $[eqn]$ based on weighted least squares optimization can be expressed as:

$$[eqn]$$

The optimized weight maps $[eqn]$ and $[eqn]$ are used to compute the final weight coefficients $[eqn]$ (Eq. 4), which are then applied to fuse the base and detail layers (Eqs. 5 and 6) to obtain the final fused image $[eqn]$ (Eq. 7).

4. Experiments

4.1. Experimental settings

We conducted extensive comparative experiments, both qualitatively and quantitatively, on the TNO [63], RoadScene [64], and LLVIP [65] datasets to demonstrate the effectiveness and robust fusion capabilities of our proposed method. Then, nine state-of-the-art (SOTA) image fusion methods are selected for comparison, including SDNet [32], ICAFusion [38], GANMcC [66], SwinFusion [67], PIAFusion [2], TarDAL [68], TGFuse [69], CrossFuse [70] and SeAFusion [71]. In the quantitative experiments, six pivotal evaluation metrics were chosen to assess the fusion performance comprehensively, including spatial frequency (SF) [72], mean squared error (MSE) [73], correlation coefficient (CC) [74], sum of the correlations of differences (SCD) [75], structural similarity index measure (SSIM) [76], peak signal to noise ratio (PSNR) [77], and Qabf [78]. All the experiments are implemented on a computer with NVIDIA GeForce RTX 3060 GPU and 16 GB memory.

4.2. Comparison experiments on the TNO dataset

First, Figs 3 and 4 give some of the visualization results on the TNO dataset. As shown in Fig 1, SwinFusion, PIAFusion, and SeAFusion cannot effectively retain the background brightness, resulting in the sky about the shape of the cloud features can not be completely presented. This is likely due to their insufficient ability to balance the intensity information between infrared and visible images during fusion, leading to the loss of low-frequency background details. TarDAL introduces some of the background noise. This noise may stem from its reliance on dense feature extraction, which can amplify irrelevant details in the background. In addition, SDNet, GANMcC, and SwinFusion are deficient in preserving significant edge features in visible images, as illustrated by the green boxes. In contrast, our method effectively avoids the above mentioned problems. This is attributed to our WLSO-optimized saliency maps, which ensure that both low-frequency background and high-frequency edge features are preserved during fusion. In Fig 2, all SOTA methods result in a complete fused image. However, in terms of details, the proposed method exhibits superior performance in enhancing texture details, allowing for a clearer depiction of fine elements such as tree branches, as indicated in the blue boxes.

Visualization results of 10 methods on “ Marne_04” image pairs in the TNO dataset.

Visualization results of 10 methods on “Kaptein_1123” image pairs in the TNO dataset.

Furthermore, the average values of six complementary metrics are listed in Table 1. Significantly, our approach demonstrates superior performance across all metrics, and one more metric achieves the second best performance. The best SF and Qabf illustrates that our fusion results achieves the most refined sharpness, which is a direct result of our emphasis on preserving high-frequency details. The best MSE also illustrates that our fusion results have better image quality. This is because our method minimizes information loss during fusion by effectively combining the base and detail layers. The excellent performance of SCD shows that our method fully complements the information in the source image. This is achieved by our visual saliency map strategy, which ensures that complementary information from both modalities is integrated seamlessly. In addition, the proposed method achieves the best SSIM and PSNR, indicating superior structural similarity and contains less noise information.

Table 1: Results of quantitative comparisons on the TNO dataset, where bold font indicates best and underlining indicates second best.

4.3. Comparison experiments on the RoadScene dataset

Fig 5 gives some visualization of the fusion results in the RoadScene dataset. It is worth noting that SDNet, ICAFusion, GANMcC, TGFuse, and CrossFuse are unable to display the background information correctly, and all methods are prone to different levels of spectral interference in the infrared imager, which is particularly evident in the discoloration of the sky background. This issue arises because these methods do not adequately handle the intensity differences between infrared and visible images, leading to improper fusion of low-frequency background regions. Furthermore, SwinFusion, PIAFusion, and TarDAL do not effectively retain critical detailed features in infrared images, as shown in the green box. On the other hand, SDNet, GANMcC, and TGFuse, in particular, are affected by the noise of the infrared image, which results in critical features not being effectively fused, and there is insufficient information retention. In contrast, our method effectively solves the above problem. This is achieved by our SVF-based decomposition, which separates noise from meaningful information. Our fused result not only correctly displays the background information, but also shows the outline of the cable completely.

Visualization results of 10 methods in the RoadScene dataset.

Similarly, Table 2 gives the results of the quantitative comparison of the 10 methods on the RoadScene dataset. It can be observed that our methods still have a competitive performance on most of the metrics. Attributed to visual saliency maps optimized by WLSO, which enables the base and detail layers to prevent information loss when key features are fused, and the maximum MSE further reflects this advantage. Also, the maximum PSNR similarly reflects that our fusion results are characterized by sharper texture details. This is because our method effectively preserves high-frequency details through the detail layer fusion process. The second-ranked Qabf illustrates the advantages of our method in preserving scene edge features and reflecting information richness. In conclusion, the comprehensive comparative results reveals that our method is able to achieve excellent fusion results.

Table 2: Quantitative comparison results on the RoadScene dataset.

4.4. Comparison experiments on the LLVIP dataset

As shown in Fig 6, we actively engaged in comparative experiments on the LLVIP dataset to showcase the superior fusion performance of our proposed method with infrared-RGB visible images. In daytime scene, the significant features of thermal radiation in the infrared image can provide more complete and complementary information for the visible image. However, SDNet suffers from the noise of infrared images, which makes the fusion quality inferior. In addition, GANMcC and TarDAL are deficient in preserving detailed features. This deficiency arises because these methods do not adequately emphasize high-frequency detail preservation during fusion. In nighttime scenes, the fused images are often susceptible to varying degrees of spectral contamination from the visible images, especially SDNet, ICAFusion, TarDAL, TGFuse, and CrossFuse, as shown in the green box. On the other hand, ICAFusion and TarDAL are also unable to completely preserve the license plate information. This is likely due to their insufficient ability to retain fine details during fusion. In contrast, our method and SeAFusion are able to completely overcome the above problems by effectively synthesizing significant thermal radiation features and texture information, without being affected by lighting conditions.

Qualitative comparison of the 10 methods on images from the LLVIP dataset.

Furthermore, Table 3 offers a comprehensive display of the objective metrics values. Obviously, our method obtains the second-best ranking performance on the majority of the metrics. Attributed to the effective SVF-based decomposition and the visual saliency measure strategy based on WLSO, the proposed method effectively preserves the texture details while highlighting the salient information, as reflected in metrics SF, CC and Qabf. From the comparative experiments, it is evident that our proposed method is characterized by its favorable fusion performance and robustness.

Table 3: Quantitative comparison results on the LLVIP dataset.

4.5. Ablation experiment

4.5.1. Effectiveness analysis of decomposition method and fusion strategy.

Ablation experiments were implemented to systematically evaluate the impact of the decomposition method and fusion strategy on the overall effectiveness of our approach. First, for the decomposition method, we chose the classical FPDE as an alternative to SVF, as shown in the group of Fig 7 (a). However, the fusion results obtained using the FPDE-based decomposition scheme are deficient in preserving texture details. In contrast, attributed to SVF for determining salient features by calculating pixel variance allow salient gradient features to be preserved. Next, for the fusion strategy, we chose a simple weighted average approach as an alternative to the WLSO-based saliency measure scheme, and the fusion results are shown in group (b) of Fig 7. It is evident that our method possesses a notable advantage in emphasizing the thermal radiation features, attributed to the superiority of the visual saliency measure in extracting the salient regions, and then after the WLSO, the resulting saliency weight map can more completely reflect the salient features in the region of interest. Moreover, Table 4 presents the objective comparison values obtained by performing the above ablation experiments on the TNO dataset. The results clearly indicate that our method holds an advantageous position in the majority of the evaluated metrics.

Table 4: Results of objective metrics on ablation studies on the TNO dataset.

Qualitative comparison results of ablation experiments on the TNO dataset, where (a) w/o SVF denotes the use of FPDE as the decomposition method and (b) w/o SW denotes the use of average method as the fusion strategy.

4.5.2. Parameter sensitivity analysis.

In addition, we conducted relevant ablation experiments on the choice of the balance coefficient $[eqn]$ in WLSO, aiming to optimize the performance of the proposed method. We performed the relevant experiments on the TNO dataset and the quantitative results are shown in Table 5. It can be found that, as $[eqn]$ increases, SF slightly decreases, indicating a reduction in high-frequency details. This is expected because a larger $[eqn]$ emphasizes smoothness in the weight map, which may suppress some fine details. MSE decreases as $[eqn]$ increases, indicating better fidelity to the source images. This is because a larger $[eqn]$ reduces noise and artifacts in the weight map, leading to more accurate fusion results. CC increases with $[eqn]$ , this is consistent with the improved fidelity observed in the MSE results. SCD decreases slightly as $[eqn]$ increases, indicating a more balanced integration of information from the source images. SSIM remains relatively stable across different values of $[eqn]$ , with a slight improvement at higher values. PSNR increases with $[eqn]$ , indicating better noise suppression and higher fidelity to the source images. Qabf increases slightly as $[eqn]$ increases, indicating better preservation of information from the source images. Based on the experimental results, we use $[eqn]$ =1 as the default value in this paper, as it provides a good balance between smoothness and fidelity, achieving high performance across multiple evaluation metrics.

Table 5: Fusion performance under different balance coefficient values λ.

4.6 Analysis of running efficiency

Running efficiency is one of the important indicators for evaluating the performance of a method. Therefore, we conducted comparative experiments on running efficiency between our proposed method and nine advanced methods on the TNO dataset, and the results are listed in Table 6. It is evident that, due to the lightweight network architecture design and GPU acceleration, the deep learning-based method SeAFusion achieved a significant advantage. Additionally, CrossFuse ranked second. In contrast, our method, based on traditional optimization theory, requires extensive mathematical calculations to compute the optimal solution, and thus does not exhibit particularly outstanding performance in computational efficiency. Therefore, in future research, how to effectively reduce the iterative computation time will be an area that needs further exploration.

Table 6: Results of runtime comparison for 40 pairs of images on the TNO dataset. (Unit: seconds).

5. Conclusion

In this paper, we proposed a novel IVIF method based on SVF and WLSO, effectively separating salient features and texture details while enhancing their visibility in fused images. Experimental results demonstrate superior performance over nine advanced methods, particularly in target highlighting and detail preservation. The method’s independence from large-scale training datasets is a significant advantage, reducing computational costs and avoiding distortion issues.

6. Limitations and future work

Despite the promising results achieved by our proposed method, several limitations should be noted. First, while our method performs well on standard datasets, it may face challenges in highly complex scenes where the distinction between salient targets and background textures is less clear. Future work could explore adaptive parameter optimization to improve performance in such scenarios. Second, the applicability of our method to other fusion tasks, such as medical image fusion or multi-spectral fusion, has not been fully explored. Future studies will investigate the generalization capability of our approach across different domains.