MMDRFuse: Distilled Mini-Model with Dynamic Refresh for Multi-Modality Image Fusion

TL;DR

This paper introduces MMDRFuse, a highly efficient, lightweight multi-modality image fusion model with a dynamic refresh training strategy, achieving high performance with only 113 parameters for real-time applications.

Contribution

The work presents a novel small convolutional network trained with a dynamic refresh strategy and multiple supervision methods, significantly reducing model size while maintaining high fusion quality.

Findings

Model contains only 113 parameters (0.44 KB).

Achieves superior fusion performance with high efficiency.

Demonstrates effectiveness in downstream pedestrian detection.

Abstract

In recent years, Multi-Modality Image Fusion (MMIF) has been applied to many fields, which has attracted many scholars to endeavour to improve the fusion performance. However, the prevailing focus has predominantly been on the architecture design, rather than the training strategies. As a low-level vision task, image fusion is supposed to quickly deliver output images for observation and supporting downstream tasks. Thus, superfluous computational and storage overheads should be avoided. In this work, a lightweight Distilled Mini-Model with a Dynamic Refresh strategy (MMDRFuse) is proposed to achieve this objective. To pursue model parsimony, an extremely small convolutional network with a total of 113 trainable parameters (0.44 KB) is obtained by three carefully designed supervisions. First, digestible distillation is constructed by emphasising external spatial feature consistency, delivering soft supervision with balanced details and saliency for the target network. Second, we develop a comprehensive loss to balance the pixel, gradient, and perception clues from the source images. Third, an innovative dynamic refresh training strategy is used to collaborate history parameters and current supervision during training, together with an adaptive adjust function to optimise the fusion network. Extensive experiments on several public datasets demonstrate that our method exhibits promising advantages in terms of model efficiency and complexity, with superior performance in multiple image fusion tasks and downstream pedestrian detection application. The code of this work is publicly available at https://github.com/yanglinDeng/MMDRFuse.