SwinFuSR: an image fusion-inspired model for RGB-guided thermal image super-resolution

TL;DR

SwinFuSR is a transformer-based model for RGB-guided thermal image super-resolution that improves image quality and robustness, outperforming existing methods in PSNR and SSIM metrics.

Contribution

The paper introduces SwinFuSR, a novel Swin transformer-based architecture for guided thermal image super-resolution with a training method for missing guidance modalities.

Findings

Outperforms state-of-the-art models in PSNR and SSIM.

Achieves 3rd place in PBVS 2024 challenge for PSNR.

Has fewer parameters than comparable models.

Abstract

Thermal imaging plays a crucial role in various applications, but the inherent low resolution of commonly available infrared (IR) cameras limits its effectiveness. Conventional super-resolution (SR) methods often struggle with thermal images due to their lack of high-frequency details. Guided SR leverages information from a high-resolution image, typically in the visible spectrum, to enhance the reconstruction of a high-res IR image from the low-res input. Inspired by SwinFusion, we propose SwinFuSR, a guided SR architecture based on Swin transformers. In real world scenarios, however, the guiding modality (e.g. RBG image) may be missing, so we propose a training method that improves the robustness of the model in this case. Our method has few parameters and outperforms state of the art models in terms of Peak Signal to Noise Ratio (PSNR) and Structural SIMilarity (SSIM). In Track 2 of the PBVS 2024 Thermal Image Super-Resolution Challenge, it achieves 3rd place in the PSNR metric. Our code and pretained weights are available at https://github.com/VisionICLab/SwinFuSR.