Fose: Fusion of One-Step Diffusion and End-to-End Network for Pansharpening

TL;DR

This paper introduces Fose, a lightweight fusion model combining one-step diffusion and end-to-end networks for efficient and high-quality pansharpening, significantly reducing computation time while improving performance.

Contribution

The paper proposes a novel four-stage training strategy to fuse one-step diffusion models with end-to-end networks, achieving faster inference and better results in pansharpening.

Findings

7.42x speedup over baseline diffusion models

Significant performance improvement on three benchmarks

Effective fusion of diffusion and end-to-end models

Abstract

Pansharpening is a significant image fusion task that fuses low-resolution multispectral images (LRMSI) and high-resolution panchromatic images (PAN) to obtain high-resolution multispectral images (HRMSI). The development of the diffusion models (DM) and the end-to-end models (E2E model) has greatly improved the frontier of pansharping. DM takes the multi-step diffusion to obtain an accurate estimation of the residual between LRMSI and HRMSI. However, the multi-step process takes large computational power and is time-consuming. As for E2E models, their performance is still limited by the lack of prior and simple structure. In this paper, we propose a novel four-stage training strategy to obtain a lightweight network Fose, which fuses one-step DM and an E2E model. We perform one-step distillation on an enhanced SOTA DM for pansharping to compress the inference process from 50 steps to only 1 step. Then we fuse the E2E model with one-step DM with lightweight ensemble blocks. Comprehensive experiments are conducted to demonstrate the significant improvement of the proposed Fose on three commonly used benchmarks. Moreover, we achieve a 7.42 speedup ratio compared to the baseline DM while achieving much better performance. The code and model are released at https://github.com/Kai-Liu001/Fose.