FusionCounting: Robust visible-infrared image fusion guided by crowd counting via multi-task learning

TL;DR

FusionCounting is a multi-task framework that simultaneously improves visible-infrared image fusion and crowd counting, especially effective in dense scenes, by leveraging joint learning, dynamic loss weighting, and adversarial training.

Contribution

It introduces a unified multi-task learning approach integrating crowd counting into VIF, with novel dynamic loss weighting and adversarial training for robustness and performance.

Findings

Enhanced fusion quality demonstrated on public datasets.

Superior crowd counting accuracy achieved.

Improved robustness against adversarial attacks.

Abstract

Visible and infrared image fusion (VIF) is an important multimedia task in computer vision. Most VIF methods focus primarily on optimizing fused image quality. Recent studies have begun incorporating downstream tasks, such as semantic segmentation and object detection, to provide semantic guidance for VIF. However, semantic segmentation requires extensive annotations, while object detection, despite reducing annotation efforts compared with segmentation, faces challenges in highly crowded scenes due to overlapping bounding boxes and occlusion. Moreover, although RGB-T crowd counting has gained increasing attention in recent years, no studies have integrated VIF and crowd counting into a unified framework. To address these challenges, we propose FusionCounting, a novel multi-task learning framework that integrates crowd counting into the VIF process. Crowd counting provides a direct quantitative measure of population density with minimal annotation, making it particularly suitable for dense scenes. Our framework leverages both input images and population density information in a mutually beneficial multi-task design. To accelerate convergence and balance tasks contributions, we introduce a dynamic loss function weighting strategy. Furthermore, we incorporate adversarial training to enhance the robustness of both VIF and crowd counting, improving the model's stability and resilience to adversarial attacks. Experimental results on public datasets demonstrate that FusionCounting not only enhances image fusion quality but also achieves superior crowd counting performance.

Tables

Table 1. TABLE I: Quantitative comparison of image fusion performance on the RGBT-CC dataset. Results in red , blue and green represent the best, the second-best and the third-best.

Methods	Qabf$\uparrow$	SSIM$\uparrow$	PSNR$\uparrow$	CC$\uparrow$	SCD$\uparrow$	SD$\uparrow$	AG$\uparrow$	SF$\uparrow$
RFN-Nest	0.57	0.64	13.86	0.44	0.72	40.37	3.63	11.43
UNFusion	0.54	0.67	14.75	0.58	1.49	27.67	2.64	7.22
Rse2Fusion	0.52	0.67	14.70	0.58	1.46	26.60	2.58	7.08
SwinFusion	0.67	0.65	13.47	0.56	1.73	44.98	4.11	11.73
PSCM	0.20	0.40	56.87	0.53	1.44	104.89	9.57	25.23
FCDFusion	0.30	0.18	10.78	0.76	1.47	43.01	3.52	10.60
IADM + FC	0.55	0.73	63.98	0.55	1.62	38.42	3.68	10.48
CSCA + FC	0.52	0.65	15.42	0.58	1.58	35.24	3.81	10.06
MC3Net + FC	0.39	0.66	15.63	0.52	1.22	33.52	3.22	9.48
DEFNet + FC	0.61	0.65	16.01	0.58	1.59	31.75	3.64	10.74

Table 2. TABLE II: Quantitative comparison of crowding counting performance on the RGBT-CC dataset.

Methods	GAME(0)$\downarrow$	GAME(1)$\downarrow$	GAME(2)$\downarrow$	GAME(3)$\downarrow$	RMSE$\downarrow$
CAG	31.38	40.28	51.04	63.52	56.18
DLF-IA	19.95	26.74	36.03	50.12	35.74
IADM	27.64	39.65	48.08	61.47	42.34
CSCA	24.17	33.81	38.29	50.09	35.4
MC3Net	21.30	30.99	41.22	56.6	32.48
DEFNet	23.35	36.97	43.3	53.54	37.89
IADM + FC	20.49	29.27	35.77	47.95	30.22
CSCA + FC	21.23	29.20	36.00	47.34	35.23
MC3Net + FC	15.84	23.96	32.5	45.56	26.02
DEFNet + FC	21.50	36.46	42.64	53.47	36.14

Table 3. TABLE III: Quantitative comparison of our FusionCounting and three representative segmentation-oriented VIF methods on the RGBT-CC dataset. Here, the FusionCounting is implemented based on MC 3 Net.

Method	SeAFusion	PSFusion	MRFS	FusionCounting
Year	2022	2023	2024	Ours
GAME(0)$\downarrow$	27.89	28.90	34.21	15.84
GAME(1)$\downarrow$	37.29	39.06	41.76	23.96
GAME(2)$\downarrow$	46.70	48.96	50.23	32.50
GAME(3)$\downarrow$	60.24	63.11	62.59	45.56
RMSE$\downarrow$	44.14	45.01	51.12	26.02

Table 4. TABLE IV: Model complexity. (model+FC)/original model is shown.

Methods	CSCA	DEFNet	IADM	MC3Net
FLOPs (G)	163.1 / 94.06	563.87 / 494.27	33.07 / 22.13	805.05 / 713.99
Params (M)	20.57 / 17.19	45.57 / 45.16	27.36 / 25.67	268.08 / 260.52

Table 5. TABLE V: Effectiveness of multi-task learning and dynamic weighting of our FusionCounting on the RGBT-CC dataset. Here, ‘ST’ represents the model only with a single task, ‘w/o DW’ represents the model of FusionCounting without dynamic weighting factors, and ‘Ours’ represents the original FusionCounting. Bold denotes the optimal.

Baselines	Settings	Visible-infrared image fusion								Crowd counting
		Qabf$\uparrow$	SSIM$\uparrow$	PSNR$\uparrow$	CC$\uparrow$	SCD$\uparrow$	SD$\uparrow$	AG$\uparrow$	SF$\uparrow$	GAME(0)$\downarrow$	GAME(1)$\downarrow$	GAME(2)$\downarrow$	GAME(3)$\downarrow$	RMSE$\downarrow$
IADM	ST	0.55	0.73	63.29	0.55	1.60	37.69	3.63	10.35	27.64	39.65	48.08	61.47	42.34
	w/o DW	0.44	0.70	63.63	0.54	1.45	34.93	3.39	9.94	22.99	33.42	40.20	52.95	35.11
	Ours	0.55	0.73	63.98	0.55	1.62	38.42	3.68	10.48	20.49	29.27	35.77	47.95	30.22
CSCA	ST	0.43	0.63	13.73	0.35	0.49	34.95	3.85	11.11	24.17	33.81	38.29	50.09	35.40
	w/o DW	0.31	0.61	13.69	0.33	0.43	31.27	3.32	8.64	24.23	33.20	38.92	50.36	38.24
	Ours	0.52	0.65	15.42	0.58	1.58	35.24	3.81	10.06	21.23	29.20	36.00	47.34	35.23
MC3Net	ST	0.55	0.69	15.60	0.33	0.43	34.79	3.67	10.23	21.30	30.99	41.22	56.60	32.48
	w/o DW	0.36	0.66	15.15	0.49	1.12	29.41	3.10	8.42	23.67	32.10	40.86	56.82	39.31
	Ours	0.39	0.66	15.63	0.52	1.22	33.52	3.22	9.48	15.84	23.96	32.50	45.56	26.02
DEFNet	ST	0.59	0.66	15.76	0.59	1.45	27.28	3.04	9.03	23.35	36.97	43.30	53.54	37.89
	w/o DW	0.42	0.60	13.55	0.31	0.32	30.65	3.58	10.61	71.70	79.26	88.32	97.41	116.31
	Ours	0.61	0.65	16.01	0.58	1.59	31.75	3.64	10.74	21.50	36.46	42.64	53.47	36.14

Table 6. TABLE VI: Results of ablation studies on the designs of multi-task learning and dynamic weighting factors of our FusionCounting on DroneRGBT dataset. Here, “ST” represents the model only with a single task, “w/o DW” represents the model of FusionCounting without the dynamic weighting factors, and “Ours” represents the original FusionCounting. Bold denotes the optimal.

Baseline	Setting	Visible-infrared image fusion								Crowd counting
		Qabf$\uparrow$	SSIM$\uparrow$	PSNR$\uparrow$	CC$\uparrow$	SCD$\uparrow$	SD$\uparrow$	AG$\uparrow$	SF$\uparrow$	GAME(0)$\downarrow$	GAME(1)$\downarrow$	GAME(2)$\downarrow$	GAME(3)$\downarrow$	RMSE$\downarrow$
IADM	ST	0.40	0.69	62.89	0.65	1.33	33.50	4.25	12.23	11.43	15.08	19.23	24.58	17.81
	w/o DW	0.38	0.69	62.89	0.65	1.37	33.34	4.01	11.44	11.43	14.10	17.86	22.71	17.44
	Ours	0.41	0.69	63.02	0.65	1.39	33.68	4.12	12.08	11.16	14.57	18.97	23.24	17.21
CSCA	ST	0.44	0.69	15.39	0.67	1.33	31.52	4.30	12.79	12.15	14.92	18.48	23.59	19.09
	w/o DW	0.38	0.69	15.60	0.67	1.38	31.61	4.04	11.57	11.67	14.10	17.66	22.44	18.20
	Ours	0.42	0.69	15.46	0.67	1.35	31.39	4.26	12.34	11.09	14.02	17.85	23.02	17.75
MC3Net	ST	0.36	0.70	15.39	0.67	1.29	30.07	3.78	10.45	8.52	10.60	13.67	18.74	13.08
	w/o DW	0.32	0.68	15.33	0.66	1.30	29.91	3.54	9.72	9.36	11.70	14.92	19.97	14.51
	Ours	0.35	0.69	15.36	0.67	1.34	30.69	3.76	10.46	8.44	10.64	13.84	19.21	12.37
DEFNet	ST	0.47	0.70	15.41	0.68	1.33	29.71	4.33	12.63	13.20	15.04	17.74	20.96	20.22
	w/o DW	0.37	0.68	15.45	0.66	1.36	32.55	3.98	11.48	11.67	14.10	17.40	20.74	19.56
	Ours	0.41	0.69	15.60	0.68	1.37	31.08	4.15	12.22	11.23	13.20	15.84	18.92	17.38

Table 7. TABLE VII: Ablation experiments on Z on the IADM model.

Z Value	GAME(0)	GAME(1)	GAME(2)	GAME(3)	RMSE
1.0	22.69	30.25	36.62	48.22	37.84
1.5	23.23	31.50	37.56	49.70	38.44
2.0	22.06	30.27	37.16	48.94	35.75
2.5	24.32	32.51	38.06	48.34	41.05
3.0	20.49	29.27	35.77	47.95	30.22

Table 8. TABLE VIII: Attack effects and adversarial training for RGBT-CC. Our input is the attacked RGBT image, and we conduct attack experiment (attack) and adversarial experiment (Advtrain) on each baseline. The input in these experiments is attacked images.

Baselines	Setting	Visible-infrared image fusion								Crowd counting
		Qabf$\uparrow$	SSIM$\uparrow$	PSNR$\uparrow$	CC$\uparrow$	SCD$\uparrow$	SD$\uparrow$	AG$\uparrow$	SF$\uparrow$	GAME(0)$\downarrow$	GAME(1)$\downarrow$	GAME(2)$\downarrow$	GAME(3)$\downarrow$	RMSE$\downarrow$
IADM	w/o Advtrain	0.27	0.49	60.72	0.25	0.31	42.72	4.68	11.22	38.63	52.50	62.28	75.99	52.92
	Ours	0.30	0.59	61.63	0.34	0.45	36.56	4.97	12.00	21.31	33.07	40.15	53.18	36.49
CSCA	w/o Advtrain	0.16	0.29	14.43	0.42	0.76	27.96	8.04	15.71	57.78	79.69	107.71	124.46	74.92
	Ours	0.30	0.62	15.59	0.56	1.54	33.19	3.01	8.81	32.95	40.70	48.22	59.45	52.32
MC3Net	w/o Advtrain	0.29	0.54	15.55	0.49	1.24	35.20	5.86	13.45	70.65	81.32	93.27	105.87	116.11
	Ours	0.30	0.66	15.67	0.51	1.29	32.64	3.09	8.31	39.08	49.14	58.96	70.91	66.57
DEFNet	w/o Advtrain	0.17	0.15	12.84	0.36	0.87	37.00	16.73	41.09	316.06	325.53	340.18	371.96	410.32
	Ours	0.32	0.63	15.84	0.58	1.52	30.58	2.74	8.37	67.73	71.31	75.67	82.55	104.93

Table 9. TABLE IX: Ablation studies on the architecture. ‘Series’ denotes the series structure of two tasks in FusionCounting.

Baselines	Structure	GAME(0)$\downarrow$	GAME(1)$\downarrow$	GAME(2)$\downarrow$	GAME(3)$\downarrow$	RMSE$\downarrow$
IADM	Series	130.33	150.96	185.96	207.96	148.03
	Ours	20.49	29.27	35.77	47.95	30.22
CSCA	Series	130.60	130.98	131.58	131.92	183.17
	Ours	21.23	29.20	36.00	47.34	35.23
MC3Net	Series	99.04	118.61	151.67	166.32	150.84
	Ours	15.84	23.96	32.50	45.56	26.02
DEFNet	Series	102.25	116.94	139.49	150.27	162.70
	Ours	21.50	36.46	42.64	53.47	36.14

Equations

$$m^k=MIX\left({\Psi }^k\left(I_{vi}^k\right),{\Phi }^k\left(I_{ir}^k\right)\right),$$

$$I_{vi}^k={\Psi }^{k-1}\left(I_{vi}^{k-1}\right)\oplus m^{k-1},$$

$$I_{ir}^k={\Phi }^{k-1}\left(I_{ir}^{k-1}\right)\oplus m^{k-1},$$

$${\mathcal{A}}^k={Conv}_{3\times 3}\left(up\left({\mathcal{A}}^{k+1}\right)\oplus m^k\right).$$

$$O_{count}={\mathcal{F}}_{Count}\left(I_{vi}^4,I_{ir}^4\right),$$

$$O_{fuison}={\mathcal{F}}_{Fusion}\left({\mathcal{A}}^1\right),$$

$${\mathcal{L}}_f={\eta }_1{\mathcal{L}}_{SSIM}+{\eta }_2{\mathcal{L}}_{text}+{\eta }_3{\mathcal{L}}_{int}+{\eta }_4{\mathcal{L}}_{MSE},$$

$${\mathcal{L}}_{SSIM}=1-\frac{1}{2}\left(SSIM\left(I_{RGB},I_T\right)+SSIM\left(I_{RGB},I_T\right)\right),$$

$${\mathcal{L}}_{text}=\frac{1}{HW}{\Vert \left|\nabla I_F\right|-\max \left(\left|\nabla I_{RGB}\right|,\left|\nabla I_T\right|\right)\Vert }_1,$$

$${\mathcal{L}}_{int}=\frac{1}{HW}{\Vert I_F-\max \left(I_{RGB},I_T\right)\Vert }_1.$$

$${\mathcal{L}}_{MSE}=\frac{1}{2}\left({\Vert I_F-I_{RGB}\Vert }_2^2+{\Vert I_F-I_T\Vert }_2^2\right).$$

$${\mathcal{L}}_c=\sum _{n=1}^{N}DF\left(1-E\left[c_n\right]\right),$$

$${\lambda }_k\left(t\right)=\frac{K}{K-1}$$

$$w_k\left(t-1\right)$$

Full text

I Introduction

Visible and infrared image fusion (VIF) [1, 2, 3, 4] refers to the integration of information from visible and infrared images to generate more informative fused images. Owing to the complementary characteristics of the information from visible images and infrared images, the fused image can be utilized to enhance the performance of various downstream applications, such as object detection, semantic segmentation, and video surveillance.

Because there are no ground-truth labels in the VIF task, most deep learning-based VIF methods are based on unsupervised learning. In these vanilla VIF methods, the loss function (Fusion loss) is obtained by calculating the difference between the generated fused image and its original visible and infrared images, as shown in Fig. 1(a). Some studies [5, 3, 6] have pointed out that optimizing fused image solely is not effective in promoting the performance of downstream vision applications. Consequently, some researchers have explored application-oriented VIF methods [5, 7, 6], i.e., consider the downstream vision application in the image fusion process. A typical example is shown in Fig. 1(b), where semantic segmentation is utilized to facilitate learning of the image fusion process.  Some other studies [7, 8] utilize object detection as the downstream application, as shown in Fig 1(c).

However, semantic segmentation tasks require labor-intensive pixel-level annotations. Although object detection reduces labeling effort compared to segmentation, it remains prone to annotation errors in highly crowded scenes due to bounding box overlap and occlusion. Moreover, most existing methods employ a cascaded structure for mutual learning between the two tasks, as shown in Figs. 1(b) and (c). This kind of design can easily lead to redundancy because both the image fusion model and the downstream application model have feature extraction steps, as shown by the two encoders in Figs. 1 (b) and (c). Furthermore, existing methods primarily focus on improving task performance. The robustness of these methods, particularly their sensitivity to environmental changes, such as those highlighted by recent focus on adversarial attacks, has been rarely addressed. Figure 2 illustrates the negative impact of adversarial attacks on the fusion results of the SwinFusion model. As shown, the structural similarity index (SSIM) of the images drops significantly following projected gradient descent (PGD) attacks, highlighting the vulnerability of traditional single-task models to such adversarial perturbations. Finally, although RGB-T crowd counting has gained increasing attention [9, 10, 11, 12], no studies have integrated VIF and crowd counting into a unified framework to explore the mutual benefits between these two tasks.

To address these issues, this paper integrates crowd counting with VIF into a unified and robust multi-task framework, as shown in Fig. 1(d). This framework leverages the synergy between VIF and crowd counting. Although crowd counting is less general than semantic segmentation or object detection, it offers significantly lower supervision costs thanks to sparse point-level annotations and fewer labeling errors. Moreover, supervision from the crowd counting task can guide the fusion process to generate images rich in semantic information, which may benefit various downstream tasks. Specifically, we leverage pedestrian semantic contexts derived from the crowd counting task to guide VIF. To enhance multi-task learning performance, we introduce a dynamic weighting strategy that improves model training. Unlike the Multiple Gradient Descent Algorithm - Upper Bound (MGDA-UB) proposed by Sener and Koltun [15], which computes a Pareto-optimal solution in the tensor space of shared parameters before applying backpropagation, our method dynamically adjusts the weights of multiple task losses and updates all model parameters in an end-to-end manner. Additionally, we design an adversarial training scheme to strengthen the robustness of our model, making it more reliable and precise against malicious attacks.

In summary, the main contributions are as follows:

• •

We propose an end-to-end multi-task learning framework for visible-infrared image fusion guided by crowd counting. To our knowledge, our FusionCounting is the first to jointly optimize VIF and crowd counting, making it particularly effective in dense crowd scenarios.

• •

We introduce a dynamic loss weighting strategy to balance the contributions of the fusion and counting tasks, accelerating the multi-task learning process.

• •

We design an adversarial training scheme to enhance the robustness of FusionCounting against adversarial attacks, ensuring reliable fusion and counting performance. This aspect is often overlooked in prior VIF work.

• •

Extensive experiments on two public datasets demonstrate the effectiveness of our method in improving image fusion quality, boosting crowd counting performance, and enhancing robustness under adversarial attacks.

II Related work

II-A Visible-infrared image fusion

Visible-infrared image fusion (VIF) [16, 17, 18, 19] has been studied for many years as VIF has a wide range of applications, such as object tracking, object objection, semantic segmentation, and salient object detection. In general, VIF can be divided into conventional methods and deep learning-based methods. Deep learning-based methods are now dominant in this field, and they can be further classified into CNN-based methods [20], GAN-based methods [21], transformer-based methods [22, 23, 24], and diffusion model-based methods [25, 26]. Traditionally, performance evaluation in VIF is usually performed in two ways, i.e., quantitative evaluation using image fusion evaluation metrics and qualitative evaluation via human observation.

II-B Application-oriented image fusion

Before 2019, VIF methods mainly aimed to generate high-quality fused images. However, because downstream applications, e.g., object detection and semantic segmentation, are not considered in the image fusion process [3, 17], these fused images may not be optimal for downstream applications. From around 2019, researchers [27] started considering downstream applications in the process of image fusion, and this kind of methods is usually called application-oriented VIF methods [3]. Since then, several downstream applications have been considered in application-oriented VIF methods, such as object detection [7], pedestrian detection [28], and semantic segmentation [5, 29, 30, 4]. However, to the best of our knowledge, crowd counting has not been applied in application-oriented VIF as a downstream application to guide the fusion process. In this work, we will fill this gap.

II-C RGB-T crowd counting

Crowd counting is a pedestrian-oriented scene understanding task. RGB-T crowd counting has gained attention due to the complementary nature of visible and infrared images, enabling more reliable predictions in complex surveillance environments. Most existing works focus on cross-modal feature fusion. For instance, Liu et al. [9] proposed an information aggregation and distribution module for adaptive feature fusion and introduce a challenging benchmark. Zhang et al. [10] and Zhou et al. [31, 11] further explored cross-modal feature aggregation using attention mechanisms to enhance counting performance. The work by Meng et al. [32] proposed a fusion-based method for multi-modal crowd counting by introducing an auxiliary broker modality, addressing ghosting artifacts in cross-modal fusion while achieving accurate counting with minimal parameter overhead.

Despite these advances, existing methods fuse cross-modal features without fusion supervision. However, crowd counting and its upstream image fusion task are strongly correlated, a relationship we aim to explore in this study. To our knowledge, this is the first work to integrate VIF into crowd counting.

III Methodology

III-A Overview of the proposed method

In this study, we propose FusionCounting, a multi-task learning framework that jointly optimizes image fusion and crowd counting to leverage their mutual benefits. Compared to semantic segmentation, crowd counting requires only point-wise annotations at pedestrian locations, significantly reducing labeling costs. FusionCounting is a general and flexible framework. In this work, we implement it based on existing RGB-T crowd counting models, enabling simultaneous learning of image fusion and crowd counting.

III-B Multi-task learning architecture

In visual information processing, the VIF model shares a highly similar structure with the RGB-T crowd counting model, as both follow a two-stream architecture. Building on this, our multi-task framework introduces a shared encoder for both tasks, integrating multiple cross-modal feature fusion modules to facilitate crowd counting in a manner similar to VIF. This shared encoder effectively extracts cross-modal feature representations while reducing redundancy in the network architecture. Task-specific decoders are then used to complete the VIF and crowd counting tasks.

Since the output size of the fusion task needs to be consistent with the input image, a series of upsampling fusion operations must be performed during the decoding process. In contrast, the crowd counting task only needs to generate a density map from the feature map to estimate the number of people, so there is no need to perform an upsampling operation. Here, we use ${I}_{vi}^{k}$ and ${I}_{ir}^{k}$ to denote the $k$-layer visible and infrared encoder inputs, ${\Psi}^{k}\left(\cdot\right)$ and ${\Phi}^{k}\left(\cdot\right)$ are $k$-layer encoders for visible and infrared input, respectively. $MIX\left(\cdot\right)$ denotes the interaction between the $k$-layer features of visible and infrared input:

[TABLE]

[TABLE]

where ${m}^{k}$ is the result of cross-modal interaction at $k$-th layer. We use $up\left(\cdotp\right)$ to denote a linear upsampling operation. The decoding result of the $k$-th layer of fusion decoder is:

[TABLE]

Note that a supplement to Eq. (3) is ${\mathcal{A}}^{4}={m}^{4}$.

Use ${\mathcal{F}}_{Count}$ and ${\mathcal{F}}_{Fusion}$ to denote decoders for the two tasks, then

[TABLE]

where ${O}_{count}$ and ${O}_{fuison}$ are crowd counting results and the fusion results, respectively.

As shown in Fig. 3, we use the encoder of an existing RGB-T crowd counting model as the shared encoder in FusionCounting. Notably, this encoder can be replaced with other crowd counting model architectures, as demonstrated in our experiments. For task-specific decoders, we retain the original regression part for crowd counting. For image fusion, we leverage hierarchical inter-layer features as decoder inputs to enable multi-scale feature representation. Since the fusion output matches the input size, upsampling layers gradually refine the inter-layer features before passing them to the image reconstruction module [13] to generate fused images.

III-C Loss function

The loss function of the proposed FusionCounting consists of two parts, i.e., image fusion loss ${\mathcal{L}}_{f}$ and crowd counting loss ${\mathcal{L}}_{c}$, and is defined as:${\mathcal{L}}_{overall}={\lambda}_{1}{\mathcal{L}}_{f}+{\lambda}_{2}{\mathcal{L}}_{c}$, where $\lambda_{1}$ and $\lambda_{2}$ are weighting parameters.

III-C1 Image fusion loss

The image fusion loss ${\mathcal{L}}_{f}$ is implemented with a combination of the $\mathcal{L}_{SSIM}$, $\mathcal{L}_{int}$, $\mathcal{L}_{int}$ , $\mathcal{L}_{MSE}$, as is expressed:

[TABLE]

where ${\eta}_{1},{\eta}_{2},{\eta}_{3},{\eta}_{4}$ are weighting parameters.

To preserve the quality of the images and measure the structural similarity between images, we use the structural similarity loss SSIM:

[TABLE]

where $F$ is the generated image, $I_{RGB}$ and $I_{T}$ are visible and infrared images, respectively. To ensure that the local texture pattern of the generated image is consistent with the target image, the texture loss is also introduced:

[TABLE]

where $\nabla$ is the Sobel gradient operator and $\left\|\cdotp\right\|$ denotes the L1 norm. To improve the accuracy of overall brightness and contrast, we also use the intensity loss for image fusion task:

[TABLE]

To accurately restore the pixel-level information of the image, we use pixel loss and calculate MSE:

[TABLE]

III-C2 Crowd counting loss

The crowd counting loss item $\mathcal{L}_{c}$ is implemented using the commonly-used Baysiean loss item [9] as:

[TABLE]

where ${c}_{n}$ is the total count associated with a given label and $E\left[{c}_{n}\right]$ is the expectation for ${c}_{n}$. $DF\left(\cdot\right)$ represents a distance function, which is ${l}_{1}$ distance in the experiments.

III-D Dynamic weighting strategy for loss functions

To enhance learning across multiple tasks, we propose a dynamic weighting strategy to balance different loss terms, inspired by [33]. Specifically, in a multi-task learning framework with $K$ tasks, each task’s loss function is assigned a weighted coefficient $\lambda_{k}$. In FusionCounting, we set $K=2$. The core idea of our dynamic weighting strategy is to accelerate the convergence of the faster-learning task, identified by greater loss variation, which reflects a higher convergence rate.

While existing approaches often reduce the weighted factor of faster-learning tasks to shift focus to other tasks, we adopt the principle of the Matthew Effect [34]. Specifically, we amplify the weighted coefficient of the faster-learning task, enabling it to converge first and simultaneously stimulate the learning of other tasks. Thus, the weighted coefficient $\lambda_{k}\left(t\right)$ of the $k$-th task in $t$-th epoch could be expressed as:

[TABLE]

where $w_{k}\left(t-1\right)$ represents the relative descending rate of $t-1$ epoch of loss values, $\mathcal{L}_{k}\left(t-1\right)$ and $\mathcal{L}_{k}\left(t-2\right)$ represent the $\left(t-1\right)$-th loss value and $\left(t-2\right)$-th loss value of the $k$-th task, $Z$ represents a factor to realize soft weighting. Similar to [35], $\frac{K}{K-1}$ is set to ensure that $\sum_{i}\lambda\left(t\right)=K$. In the detailed implementation, $w_{k}\left(t\right)$ is set to 1, when $t$ equals to 1 and 2; the parameter $Z$ is set to 3.

The above multi-task training algorithm with the developed dynamic weighting strategy is provided in Algorithm 1

III-E Adversarial training scheme

To improve robustness, we develop an adversarial training scheme for the proposed method. This scheme integrates samples generated by PGD attacks [14] into the training set, allowing the model to learn and adapt to such attack patterns. As a result, it strengthens the model’s defense against malicious attacks and enhances overall robustness. The adversarial training algorithm is detailed in Algorithm 2.

III-F Model inference

In the inference stage, image fusion results are provided by the fusion decoder, while crowd counting results are provided by the counting decoder. This is different from most existing application-oriented VIF methods (e.g., SeAFusion [5] and SegMiF [29]), where the results of downstream applications, such as segmentation and detection, are obtained by running another segmentation or detection model on the fused images.

IV Experiments

IV-A Experimental settings

IV-A1 Implementation details

Our FusionCounting is implemented in Pytorch on a NVIDIA GeForce RTX 4090 GPU. Adam optimizer is used to update parameters and cosine annealing algorithm is used to adjust the learning rate to ensure the stability and convergence speed of the training process. The learning rate is 1$\times$10*-5*. ${\eta}_{1},{\eta}_{2},{\eta}_{3},{\eta}_{4}$ in Eq. (5) are 20, 30, 30, and 100, respectively. In addition, since there is no universally fixed configuration for PGD, we follow the principles outlined in the paper to balance attack strength and imperceptibility. Specifically, the maximum perturbation parameter $\epsilon$, step size $\alpha$, and number of iterations are set to 20, 5, and 7, respectively.

IV-A2 Datasets

We evaluate the developed FusionCounting on the following two widely-used datasets: RGBT-CC [9] and DroneRGBT [36]. RGBT-CC is a public cross-modal visible-infrared crowd counting dataset captured from a surveillance perspective. Since its original samples are not strictly aligned, this study selected a subset with aligned data to meet the requirements of the joint image fusion task, consisting of 176 RGB-T image pairs for training, 23 for validation, and 109 for testing. Compared to the original test set with an average crowd count of 74.12, our selected subset has a significantly higher average of 130.6, indicating denser and more challenging scenes that may lead to performance drops. DroneRGBT is a visible-infrared crowd dataset captured from a drone perspective, consisting of 3,600 image pairs with a 50% training and 50% test split. To monitor overfitting, we used 10% of the training set as a validation set, which led to a slight decrease in training performance.

IV-A3 Evaluation metrics

VIF metrics. For the image fusion task, eight commonly-used evaluation metrics were selected: metrics based on information entropy (PSNR [37] and Qabf [38]), metrics based on correlation (CC [39] and SCD [40]), metrics based on structural similarity (SSIM [41]) and image features metrics (AG [42], SF [42], SD [40]). These evaluation metrics comprehensively evaluate the quality of fused images from multiple aspects, ensuring the objectivity and reliability of the evaluation results.

Crowd counting metrics. Following existing studies [43], we use the classical Root Mean Square Error (RMSE) and the Grid Average Mean absolute Error (GAME) [44] as evaluation metrics. For the GAME calculation, the density map is divided into non-overlapping $4^{l}$ regions, and the error is calculated separately in each region. Lower GAME($l$) values represent better crowd counting performance.

IV-A4 Compared methods

VIF methods. We selected nine state-of-the-art VIF methods for comparison, i.e., RFN-Nest [45], SwinFusion [13], Rse2Fusion [46], UNFusion [47], PSCM [48], FCDFusion [49], SeAFusion [5], PSFusion[50], MRFS [6]. In these methods, SeAFusion, PSFusion and MRFS are application-oriented VIF methods.

Crowd counting methods. We compared our method with six crowd counting methods, i.e., IADM [9], CSCA [10], MC3Net [11], DEFNet [31], CAG [12], DLF-IA [51]. These methods take RGB-T image pairs as input.

IV-B Results of visible-infrared image fusion

Qualitative results. The qualitative comparison of image fusion performance is shown in Fig. 4. As can be seen, most methods, including FCDFusion, RFNFusion, SwinFusion, UNFusion, and PSCM, have significant color distortion in the fused image. Though the fused image of Res2Fusion keeps color information in a good way, the fused results present more blurring than ours.

Quantitative results. The quantitative image fusion results are shown in Table I. We implemented our FusionCounting based on several crowd counting methods, i.e., IADM [9], CSCA [10], MC3Net [11], and DEFNet [31]. The results show that FusionCounting achieves highly competitive performance across several metrics. This is facilitated by the mutual reinforcement of the two types of supervision, which reflects the advantages of multi-task supervision. For example, when IADM is integrated into our FusionCounting framework, the best SSIM and PSNR results are obtained. It is worth noting that our method is trained using much less RGB-T image pairs than the compared VIF methods.

IV-C Results of RGB-T crowd counting

IV-C1 Comparison with RGB-T crowd counting methods

Qualitative results. Qualitative comparison of crowd counting results is shown in Fig. 5. As can be seen, both predicted density map and counting results given by FusionCounting are very close to the ground truth. In contrast, the compared methods show larger difference with the ground-truth counting result. This is attributed to the benefit of the VIF task on the crowd counting task in our FusionCounting.

Quantitative results. Table II reports the quantitative comparison of our FusionCounting with the recent representative RGB-T crowd counting methods on RGBT-CC. It can be clearly seen that our FusionCounting shows significant competitive advantages over various metrics for the crowd counting task. More importantly, compared with the basic version of IADM, CSCA, MC3Net and DEFNet, the proposed FusionCounting built on these methods achieves significant improvements. Although DEFNet+FC appears less competitive than other FC-based models, it still clearly outperforms the original DEFNet and several classical baselines, demonstrating the robustness of our FC module. Moreover, compared with CAG and DLF-IA methods, our FusionCounting also shows an improvement in a large margin. These results further demonstrate the benefits of our multi-task learning framework on the crowd counting task.

IV-C2 Compared with application-oriented VIF methods

We compared our FusionCounting with three representative application-oriented VIF methods: SeAFusion [5], PSFusion [50], and MRFS [6]. First, we applied these methods to the RGBT-CC dataset to generate fused images, then used MC3Net for crowd counting. The results, shown in Table III, indicate that FusionCounting outperforms these three methods. This is because these methods were jointly trained for image fusion and semantic segmentation, making them less effective at learning features beneficial for crowd counting. In contrast, FusionCounting directly learns crowd counting-relevant features through joint training of VIF and crowd counting. These findings suggest that features learned via semantic segmentation do not generalize well to crowd counting, highlighting the necessity of jointly training VIF with crowd counting to obtain good crowd counting performance.

IV-C3 Comparison of parameters

Table IV reports the model complexity of different baselines with and without the proposed FusionCounting module. The results show that adding FusionCounting moderately increases FLOPs and parameter counts across all methods, while maintaining a reasonable computational overhead.

IV-D Ablation studies

IV-D1 Effectiveness of multi-task learning

Tables VI and V present the results of ablation studies on multi-task learning on two datasets. In the table, ‘ST’ indicates that the model is trained for a single task (either VIF or crowd counting), i.e., only fusion decoder or counting decoder is utilized. Comparing the single-task settings with the original setting (denoted as ‘Ours’), it can be observed that the proposed FusionCounting significantly enhances both image fusion and crowd counting results, regardless of the baseline used. This improvement is attributed to the mutual benefits between VIF and crowd counting in our FusionCounting framework.

IV-D2 Effectiveness of dynamic weighting factors

Table VI and V show the effectiveness of dynamic weighting factors. In the table, “w/o DW” refers to variants without dynamic weighting factors. Comparing the results of FusionCounting without dynamic weighting factors (w/o DW) to the original setting (Ours), it can be observed that the proposed dynamic weighting factors significantly improve both VIF and crowd counting performance. Figure 6 shows the comparisons of the training loss curves with and without dynamic weighting factors. It can be observed that the loss decreases significantly faster when using dynamic weighting factors compared to the case without them during the initial stage of training. This improvement can be attributed to the effective design of the weighting strategy, which balances the dependence between tasks. This observation also indicates that the introduction of dynamic weighting factors can more effectively guide the model to rapidly converge to the optimal solution.

Additionally, in Table VII, ablation experiments with parameter Z are shown on the model IADM, with values 1 to 3, and the best index is achieved when the parameter is 3.

IV-D3 Effectiveness of adversarial training

Table VIII presents the results of ablation study on adversarial training. Here, the input is attacked images. It can be seen that introducing the adversarial learning makes our FusionCounting more robust to the adversarial attacks no matter what is the baseline. Moreover, both the VIF and crowd counting performance is more robust after adversarial training.

Robustness analysis. As shown in Fig. 7 and Fig. 8, adversarial attacks disrupt the density maps and significantly reduce the SSIM of visible and infrared images, leading to degraded fusion quality with increased noise and blur. In contrast, adversarially trained models produce density maps closer to the ground truth and improve fusion SSIM, resulting in clearer, less noisy images. These results demonstrate the effectiveness of adversarial training in enhancing the robustness and anti-interference capability of the model.

IV-D4 Effectiveness of the parallel architecture

To verify the effectiveness of our parallel architecture, we also train a model using a serial structure for the VIF and crowd counting tasks—where the fusion decoder and counting decoder are cascaded. The comparison results are presented in Table IX. As shown, the parallel structure of FusionCounting significantly benefits the downstream crowd counting task. This suggests that learning both tasks concurrently allows better feature sharing and reduces error propagation between stages.

IV-E Discussions

Tables V and VI show that the multi-task learning framework not only enhances image fusion quality but also improves counting performance through shared feature representation. As acknowledged, the counting head participates in training, sharing the backbone encoder with the fusion task—a standard practice in multi-task learning where auxiliary tasks provide additional supervision to guide feature learning and improve generalization. Specifically, within this shared encoder, features extracted at different levels capture local details, structural patterns, and global semantics. Low-level features preserve image edges and suppress noise, while high-level features provide abstract semantic information crucial for the counting task. The feature fusion module then gradually integrates these multi-level features, forming a richer and more comprehensive representation. This hierarchical fusion mechanism enhances information sharing and interaction between the tasks, allowing them to mutually reinforce each other. Importantly, while the counting head introduces valuable inductive bias during training, it does not directly contribute features or outputs to the fusion head, nor is it utilized during inference for the fusion task. Therefore, this design adheres to the multi-task learning paradigm, providing justified auxiliary supervision without introducing unfair advantage or information leakage.

In summary, multi-task learning enables the fusion and counting tasks to complement each other effectively, providing a robust joint learning paradigm where the shared representation benefits both tasks.

V Conclusions

In this study, we propose FusionCounting, a multi-task framework that connects two tasks in parallel through a shared encoder, enabling mutual reinforcement and optimizing both visible-infrared image fusion and crowd counting performance. To further improve performance, we design a dynamic loss weighting strategy that adaptively adjusts the weight ratio of current losses, accelerating network convergence and improving performance. Additionally, to enhance the robustness of the network, we incorporate an adversarial training strategy. By learning from attack data, the network mitigates its negative impact, strengthening the overall robustness of the multi-task framework. In future work, we will focus on addressing the misalignment between visible and infrared images, which remains a key challenge in cross-modal crowd counting. Our next step is to develop advanced alignment modules or explore alignment-robust representations, aiming to further improve the performance and practicality of FusionCounting.